OFFICE OF APPLIED STUDIES 


Drug Abuse Warning Network: 
Development of a New Design 

Methodology Report 


DEPARTMENT OF HEALTH AND HUMAN SERVICES 
Substance Abuse and Mental Health Services Administration 

www.samhsa.gov 


Development of a New Design for DAWN 



ACKNOWLEDGMENTS 


This report was prepared for the Substance Abuse and Mental Health Services Administration 
(SAMHSA), Office of Applied Studies (OAS), by Westat under Contract Nos. 282-98-0015 
(Tasks 11 and 25) and 282-98-0003. Dr. Lori Ducharme, Dr. Caroline McLeod, and Jim Green 
(Westat) and Al Woodward (SAMHSA/OAS Project Officer for the Alternative Design of the Drug 
Abuse Warning Network [DAWN]) wrote the report. Other significant contributors include Dr. 
Judy Ball (SAMHSA/OAS Project Officer for Statistical Reports from DAWN) and Dr. Wendy 
Kissin (Project Director) and Tracy Garfield (Westat). 


PUBLIC DOMAIN NOTICE 

All material appearing in this publication is in the public domain and may be reproduced or 
copied without permission from the Substance Abuse and Mental Health Services 
Administration (SAMHSA). However, this publication may not be reproduced or distributed for a 
fee without the specific, written authorization of the Office of Communications, SAMHSA. 
Citation of the source is appreciated. Suggested citation: 

Substance Abuse and Mental Health Services Administration, Office of Applied Studies. 
Drug Abuse Warning Network: Development of a New Design (Methodology Report). 
DAWN Series M-4, DHHS Publication No. (SMA) 02-3754, Rockville, MD, 2002. 


OBTAINING ADDITIONAL COPIES OF PUBLICATION 

Copies may be obtained, free of charge, from the National Clearinghouse for Alcohol and Drug 
Information (NCADI). The NCADI is a service of the Substance Abuse and Mental Health 
Services Administration (SAMHSA). Write or call NCADI at: 

National Clearinghouse for Alcohol and Drug Information (NCADI) 

P.O. Box 2345, Rockville, MD 20847-2345 
301-468-2600 800-729-6686 TDD 800-487-4889 


ELECTRONIC ACCESS TO PUBLICATION 

This publication can be accessed electronically through Internet World Wide Web connections: 

http://www.samhsa.gov 

http://www.drugabusestatistics.samhsa.gov 


ORIGINATING OFFICE 

SAMHSA, Office of Applied Studies 
5600 Fishers Lane, Room 16-105 
Rockville, MD 20857 

August 2002 


2 


Development of a New Design for DAWN 



CONTENTS 


Section Page 

Acknowledgments. 2 

Introduction. 7 

1. Development of an Alternative Design for DAWN. 11 

2. Assessing the Impact of Health System Change on DAWN. 37 

3. Development of a New Sample Design for DAWN. 63 

4. Redesigning DAWN’S Case Definition, Data Elements and Case 

Screening Procedures. 99 

LIST OF TABLES 

Table Page 

1-1 Metropolitan Areas in Proposed Sample Redesign. 20 

1- 2 Recommended Data Elements for DAWN (ED Component). 25 

2- 1 Trends in ED Utilization, 1993-97 (NHAMCS data). 45 

2-2 ED Cases with Patient as Expected Source of Payment, 1993 and 

1997. 46 

2-3 Trends in Urgent/Emergent ED Visits, 1993-1996 . 47 

2-4 Use of Ambulatory Care by Insurance Status, 1996. 48 

2-5 Urgent/Emergent ED Visits by Type of Insurance. 50 

2- 6 ED Utilization by HMO Status for Insured Persons, 1987 and 1996 

(MEPS). 51 

3- 1 Population and Sample Sizes by MSA, 2000. 68 

3-2 Precision Requirements for Original Sample Design. 69 

3-3 Current Performance (RSEs) by MSA, 1999. 69 


Development of a New Design for DAWN 


3 




















CONTENTS 


LIST OF TABLES (continued) 

Table Page 

3-4 Geographic Coverage Options Considered in DAWN Sample 

Redesign. 72 

3-5 Stratification by Size. 80 

3-6 National Estimate-Only Scenarios. 81 

3-7 Actual or Minimum Required Responding Sample Size by Design 

and MSA Precision Level. 82 

3-8 Minimum Sample Sizes Required by Stratum and Estimates, 

Selected MSAs. 84 

3-9 Estimated Minimum Required Responding Sample Sizes for Last 

Round of Scenarios (includes national panel sample size). 85 

3-10 Estimated National Panel Sample Sizes for Last Round of Scenarios 86 

3-11 Actual Expected RSEs for the Proposed New Design. 86 

3-12 Direct Reimbursement Model Coefficients by Facility Location. 87 

3-13 Cost-Minimization Results. 89 

3- 14 Minimum, Mean and Maximum R-square Measure for Two-Phase 

Models. 92 

4- 1 Recommended Data Elements for DAWN (ED Component). 108 

4-2 Ten Most Common Diagnoses in Drug-Related Cases (Not mutually 

exclusive). Ill 

4-3 Categories Included in Current and Proposed Case Definitions. 111 

4-4 “Type of Case” Results from Field Study. 112 

4-5 Selected Conditions Associated with Cases. 113 

4-6 Disposition of Cases. 113 

4-7 Problematic Data Elements. 114 


4 


Development of a New Design for DAWN 




















CONTENTS 


LIST OF TABLES (continued) 

Table Page 

4-8 Characteristics of NEISS Cases Meeting the Proposed Case 

Definition. 115 

4-9 Number of New Cases by Size of Hospital. 116 

4-10 Estimated ED Visits per Reportable Case (NEISS Data, July-August, 

2000). 117 

4-11 Sensitivity and Specificity of Log Screen Used in Field Test (Current 

Case Definition). 120 

4-12 Sensitivity and Specificity of Log Screen Used in Field Test (New 

Case Definition). 121 

4-13 Performance of ICD-9-CM Screen (Current Case Definition). 123 

4-14. Performance of ICD-9-CM Screen (New Case Definition). 124 

LIST OF FIGURES 

Figure Page 

1-1 Type of Case Classification. 22 

3- 1 DAWN Sample Design Scenarios Evaluation System. 79 

4- 1 Types of Case Classification. 106 


Development of a New Design for DAWN 


5 













6 


Development of a New Design for DAWN 



INTRODUCTION 


It will be a continuing and dynamic effort, which will be constantly reviewed and modified 
to obtain more precise information (Bartels, 1973). 


But 


T his was the vision articulated for the Drug Abuse Warning Network (DAWN) upon 

publication of its first findings in 1973. This is also the vision of DAWN we hold today, 
for many of the intervening 30 years, DAWN failed to live up to its promise. This 
publication contains a series of reports and analyses developed as part of a two-year evaluation 
of design alternatives, which has returned DAWN to its original vision and has forged for DAWN 
a new set of purposes and approaches for the 21 st century. 


The Drug Abuse Warning Network (DAWN) is an important source of national and local 
information on substance abuse. This information is derived from data on visits to hospital 
emergency departments (EDs) and drug-related deaths reviewed by medical examiners and 
coroners (ME/Cs). DAWN collects data on the demographic characteristics of substance 
abusers and the specific drugs involved in each drug-related ED visit or death. The detail 
available on specific drugs is not matched by any other data system. Currently, DAWN is 
managed by the Office of Applied Studies (OAS), a component of the Substance Abuse and 
Mental Health Services Administration (SAMHSA) within the U.S. Department of Health and 
Human Services. SAMHSA is directed under Section 505 of the Public Health Service Act (42 
U.S.C. 290aa-4(c)(1) to collect such data. 

Defining the purpose(s) of DAWN was central to redesigning the system. The redesign 
process began formally on November 10, 1997, when OAS convened an expert panel to 
consider the future of DAWN and the need for a new design. Perhaps the most obvious and the 
most critical purpose of DAWN is its warning capacity. DAWN is intended to serve as a first 
indicator of the serious consequences of drug use in the 21 metropolitan areas served by 
DAWN. During its three decades of operation, however, this purpose has been met 
inadequately at best. 

DAWN data are used for many other purposes with varying degrees of success, 
appropriateness, and satisfaction. The purposes for which DAWN is well suited include drug 
scheduling and drug labeling, quantifying the extent of the nation’s drug problem, assessing 
effectiveness of local anti-drug efforts, guiding resource allocation decisions, surveillance of 
local area drug trends, documentation of drug problems and trends, and as a data source for 
academic research on drug abuse. Often, however, individuals used DAWN for purposes 
contrary to its design, and DAWN was frequently criticized for failing to meet such purposes. 


DAWN’s inability to fulfill its primary purposes reflects more than a mere misunderstanding 
of what DAWN was designed to accomplish. During its three decades of operation, much of the 
original structure and many of the operations continued much as they had at the beginning, 
even though the needs of the data users and the health care environment had changed. 

Clearly, technological barriers prevented DAWN from realizing its warning purpose in the 1970s 
and 1980s. However, when technologies to enable rapid data collection and dissemination 
became the norm, DAWN continued to use paper forms for data collection. The time required to 
collect, process, and analyze data collected in this way defeated any hope of realizing an early 
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warning capability, and dissemination of information back to communities was simply not a high 
priority. 

Mindful of these limitations, SAMHSA convened the 1997 review panel to consider the 
utility of DAWN and whether changes in the health care delivery system, particularly the growth 
of managed care, had diluted the value of emergency departments as a setting for collecting 
data on drug abuse. The panel, which could have called for DAWN’S elimination, recommended 
instead that SAMHSA update its procedures and prepare DAWN for the 21 st century. 

In response, OAS undertook the comprehensive two-year evaluation of DAWN. This 
evaluation resulted in recommendations for a new design for DAWN. Using a wide range of 
study methods, the evaluation addressed the following questions: 

■ Who are the users of DAWN data and what information do they need? 

■ Does DAWN collect data from the right settings? 

■ Does DAWN collect data on the right set of patients? 

■ Does DAWN collect the right data on those patients? 

■ How can DAWN use technology to collect data more efficiently? 

■ How can DAWN deliver information more effectively? 

These questions challenged the fundamental approaches long used by this survey. This 
publication presents some of the findings used to improve DAWN’S design, guided by these 
important questions. 

The first chapter discusses the development of an alternative design to DAWN, including 
limitations of the current design, the scope of the redesign activities, and key issues. The 
second chapter assesses the impact of health system change on DAWN, through a 
comprehensive review of the literature, analyses of national data, and qualitative research on 
local health system impacts. The third chapter discusses the development of a new sample 
design for DAWN. Finally, the fourth chapter discusses the redesign of DAWN’S case 
definitions, data elements, and case screening procedures. 
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1. DEVELOPMENT OF AN ALTERNATIVE 
DESIGN FOR DAWN 


T he Drug Abuse Warning Network (DAWN) was implemented nearly 30 years ago to 
support a specific set of Federal policy-making purposes. Over time, the agencies 
responsible for the maintenance of DAWN have changed, operational approaches have 
been modified, and the uses for DAWN data have expanded. First administered by the Drug 
Enforcement Administration (DEA) and the National Institute on Drug Abuse (NIDA), the 
responsibility for DAWN now rests with the Office of Applied Studies (OAS) of the Substance 
Abuse and Mental Health Services Administration (SAMFISA). In the 1980s, improvements in 
the sampling design were made to accommodate both local (metropolitan area) and Federal- 
level needs for data. In addition, the hospitals in the sample were assigned weights to permit 
the calculation of national estimates of drug-related emergency department visits and specific 
drug mentions. At the same time, the health care system has changed dramatically, as have 
the needs of DAWN’S users. Flowever, the day-to-day operations of DAWN have changed 
remarkably little over the past 3 decades. 

With the exception of the sample design and estimation procedures (described in detail in 
SAMFISA, 1998), the history of DAWN is neither well documented nor widely understood. Many 
of the current design features of the system are historical artifacts (e.g., certain data elements, 
the oversampled metropolitan areas, etc.), but there are many who incorrectly assume that 
these features reflect a specific rationale or intent on the part of OAS. The complexities and 
limitations of the system, coupled with lack of detailed historical documentation, lead to 
inevitable misinterpretation of DAWN data, as well as criticism that sometimes is well grounded 
and sometimes is not. Through an extensive DAWN redesign process, OAS sought to improve 
both the structure of the system and the public perception of DAWN. 


LIMITATIONS OF THE CURRENT DESIGN 

The value of DAWN for Federal and local policy-making is considerable but not well 
advertised, while the limitations of the system are often overemphasized by constituencies 
DAWN was not designed to serve. In 1995, the Rand Corp. published a report articulating 
many of the criticisms that have been leveled against DAWN in recent years (Caulkins, Ebener, 
& McCaffrey, 1995). That report was soon followed by a 1997 review panel, convened by OAS 
to assess the value and future prospects for DAWN. Those reviews highlighted the problems 
needing to be resolved with DAWN if the system is to achieve what was thought to be its 
considerable potential. 

Caulkins et al. identified a number of problems, limitations, and misconceptions about 
DAWN, including the following: 

■ There was little public documentation of DAWN’s data quality, and a few published 
studies raised concerns that measurement error in DAWN was substantial and 
systemic. 
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■ Because DAWN monitors episodes, not individuals, data are misinterpreted if analysts 
view them as prevalence measures. 

■ There is an uncertain, and perhaps inconsistent, relationship between the number of 
DAWN episodes and the true level of drug abuse problems in a given area. 

■ Delays in reporting DAWN data undermined its potential utility as an early warning 
mechanism for emerging drug problems. 

■ DAWN data were often assumed to represent heavy or chronic drug users, although 
the validity of that assumption has never been assessed. 


The report concluded that DAWN was used for purposes it was not designed to serve, while 
at the same time its strengths were not fully appreciated or taken advantage of. 

In 1997, OAS convened two review panels to consider the future of DAWN and the need for 
a new design. The first panel, convened to consider the uses and limitations of DAWN, 
consisted largely of Federal staff from OAS, the Food and Drug Administration (FDA), DEA, and 
the Office of National Drug Control Policy (ONDCP); in addition, two epidemiologists, a medical 
examiner, and several drug abuse researchers contributed to the group’s deliberations. The 
second panel, convened to consider the implications of health system change for DAWN, 
consisted of representatives from several managed behavioral health care organizations, a drug 
abuse treatment provider, an emergency physician, and drug abuse and health system 
researchers, along with staff from OAS and the Agency for Health Care Policy and Research 
(AFICPR, now the Agency for Flealthcare Research and Quality [APIRQ]). Each group gathered 
for a full-day meeting to identify the strengths and limitations of DAWN and to prioritize redesign 
efforts. 

Importantly, the two groups agreed that DAWN was a valuable system that, with certain 
changes, would have much to offer the policy, programming, and research communities. 
Plowever, it was clear that there was substantial work to be done to develop and implement a 
new design. Together, the two groups identified the following issues that needed to be 
addressed in a redesign of DAWN: 

■ DAWN suffered from substantial report production lags, such that data were often too 
old to be useful by the time reports were published. Flow could DAWN be made more 
timely? 

■ DAWN was intended to be a warning system but generally failed to achieve that 
function. Aside from the timeliness of published reports, how could DAWN more 
rapidly gather, identify, and disseminate information about emerging drug trends? 

Could DAWN effectively serve as a leading indicator of drug abuse trends? 

■ DAWN’S utility seemed to be under-appreciated at the local level. Were there local 
audiences for DAWN data, and how could DAWN be designed to better meet their 
needs? 

■ There was little feedback of data or information to participating facilities, which affected 
participation, recruitment efforts, and a sense of investment in the system. Flow might 
DAWN provide useful, timely feedback to facilities and communities? 
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■ The U.S. healthcare system had undergone substantial changes since DAWN’S 

inception in 1972. What impact had ongoing changes in the health care system had on 
DAWN’S ability to gather complete, accurate, and valid data from emergency 
departments? To what extent were emergency department (ED) data appropriate for 
identifying meaningful changes in drug abuse patterns? 

In addition, it was clear from the transcripts of these meetings that DAWN’S case definition 
(the rule used to define reportable cases) was difficult to explain, and therefore led to substantial 
confusion about what DAWN measured, the types of cases considered reportable, and how the 
data should be interpreted. Thus, another issue was how the system might be redesigned to 
ensure that the data were responsive to users’ needs, able to be gathered consistently, and 
readily interpretable. 


SCOPE OF THE REDESIGN ACTIVITIES 

The DAWN Redesign task order grew out of the ideas and concerns expressed in the 
Caulkins et al. report and in the two OAS review panel meetings. The original task order posed 
the following questions: 

(1) How can DAWN be made more useful in terms of predicting the patterns, trends, and 
consequences of substance abuse? 

(2) How can the quality and timeliness of DAWN data collection and reporting be 
improved? 

This chapter provides an overview of the activities undertaken to address to those 
questions, and the recommendations stemming from those activities. It should be noted that 
this chapter addresses design recommendations - that is, what changes should be made - and 
focuses less on specific methods for implementation of those recommendations. Some 
changes (e.g., improvements in the timeliness of published reports) were made by the DAWN 
team independent of the redesign activities; other changes (e.g., a redesign of the Medical 
Examiner report) were implemented by the redesign team; still other changes (e.g., 
implementation of an on-line sentinel event network) were beyond the scope of the redesign 
contract and will be the responsibility of the contractor awarded the operation of the new DAWN. 


What is DAWN? 

The Drug Abuse Warning Network is an ongoing, national data system that collects and 
reports information on adverse health consequences associated with drug abuse. Specifically, 
DAWN gathers data on drug abuse-related ED visits from a representative sample of hospitals 
in the coterminous United States, as well as data on drug abuse-related deaths reviewed by 
participating medical examiners and coroners (ME/Cs). DAWN is an important indicator of the 
nation’s drug abuse problems. The structure, content, and operation of each component are 
briefly reviewed here. 

DAWN collects data from a representative sample of 24-hour EDs operating in non-Federal, 
short-stay general medical/surgical hospitals throughout the coterminous United States. In 
2000, about 470 EDs participated in DAWN. The ED sample is structured such that DAWN can 
provide estimates of the total number of drug abuse-related ED visits (“episodes”) and the total 
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number of drug mentions for each of 21 selected metropolitan areas as well as the Nation. 
Because no attempt is made to identify unique individuals within or across EDs, the number of 
DAWN episodes will equal or exceed the number of individual persons treated for drug-related 
complications in the Nation’s EDs. 

DAWN also collects data from ME/Cs. In 2000, about 140 jurisdictions in 43 metropolitan 
areas provided data to DAWN. Each metropolitan area covers one or more death investigation 
jurisdictions. Coverage of jurisdictions varies from one metro area to the next. In a few areas, 
all jurisdictions report to DAWN; in other areas, less than half of all jurisdictions participate. 
Because the participating jurisdictions are not based on a statistical sample, their data cannot 
be used to generate estimates of the total number of drug-related deaths, either nationally or for 
the metropolitan areas. 

In each participating facility, a designated DAWN “reporter” is asked to review the charts of 
all patients treated in the ED or decedents reviewed by the ME/C. The content and quality of 
patient/decedent charts vary from one facility to the next. ED charts generally include 
information on the presenting complaint, the nurse or physician’s assessment, diagnosis, 
medications, and discharge status. ME charts include information pertinent to the death 
investigation, which may involve crime scene reports, police reports, interviews with family and 
friends, and autopsy results. Both ED and ME charts commonly contain the results of 
toxicology tests conducted as part of the assessment/ investigation process. Based on their 
retrospective review of available information, DAWN reporters are asked to identify cases in 
which the ED visit or death was induced by or related to drug abuse. 

For the purposes of DAWN, the term “drug abuse” applies if the following conditions are 

met: 


(1) The case involved at least one of the following: 

■ Use of an illegal drug; 

■ Use of a legal drug contrary to directions; or 

■ Inhalation of a nonpharmaceutical substance 

and 

(2) The substance was used for one of the following reasons: 

■ Because of drug dependence, 

■ To commit suicide (or attempt to commit suicide), 

■ For recreational purposes, or 

■ To achieve other psychic effects. 

For each reportable case, the DAWN reporter provides demographic information on the 
person involved and identifies the abused drug(s) noted in the chart. Route of administration is 
also collected for each drug reported. For ED episodes, information about the reason for the 
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ED visit, the reason for using the substance, the form and source of the substance, and the 
patient’s disposition are also collected. For ME/C cases, reporters record whether the case was 
determined to be drug-induced or drug-related, the manner of death, as well as information 
available for determining the reportability of the case (e.g., autopsy results, toxicology reports, 
death investigation notes, etc.). 

For ED episodes, up to four drugs may be reported; for ME/C cases, up to six drugs may be 
reported. Alcohol involvement is reported only for cases meeting the above criteria (i.e., cases 
involving only alcohol and no other drug are not reportable). Each drug reported for a given 
case is called a “mention.” Because each case may have multiple drug mentions, the number 
of mentions always exceeds the total number of DAWN cases. 

DAWN data are notable for the richness of detail they provide relative to other available 
data systems. Other data systems include the following: 

■ National Hospital Ambulatory Medical Care Survey (NFIAMCS), operated by the 
National Center for Health Statistics, which collects data on ED and hospital outpatient 
visits; 

■ Consumer Product Safety Commission’s National Electronic Injury Surveillance 
System (NEISS), which collects data on injuries treated in EDs, some of which are 
drug-related; and 

■ Mortality data from the National Vital Statistics System, which include cause-of-death 
data as documented on death certificates. 

Both the DAWN ED and ME data gather very specific drug information at a level of detail 
not available from other sources, which tend to rely on International Classification of Disease 
(9 th or 10 th revision [ICD-9/10]) codes to identify drug-related cases. ICD-9/10 codes have only 
a few specific drug codes (e.g., for heroin, cocaine, benzodiazepines) and relegate all others to 
a general “not otherwise classified” category. Analysts interested in obtaining data on a specific 
drug outside these categories (e.g., Ketamine) cannot do so with other data systems. 

Moreover, there are variations in the ways in which ICD-9/10 codes are assigned. In EDs, ICD- 
9-CM codes are often assigned as part of the billing function; constraints on insurance coverage 
for drug-related cases may affect the extent to which drug-related ICD-9-CM codes are actually 
assigned to cases that could be otherwise coded. Vital statistics mortality data, which are 
coded from death certificates (which are themselves a summary of the decedent’s case file), 
may assign ICD-10 codes to the underlying cause of death, missing the contributory effects of 
drug abuse. 

It is the richness of the DAWN drug vocabulary that makes the system of unique and 
important value to a wide range of agencies and organizations concerned with drug abuse 
epidemiology, prevention, treatment, and control. In the next section, the primary users and 
uses of DAWN data are reviewed. Throughout the redesign process, these constituents and 
their data needs were a primary concern, and recommendations on revisions to DAWN carefully 
considered the likely impact on these audiences. 
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Users and Uses of DAWN 


Development of an alternative design for DAWN began with a thorough assessment of the 
current and potential users of DAWN and the applications they make, or desire to make, of the 
data. This assessment of DAWN’S utility covered a broad range of topics, including the 
following: 

■ Exploration of the specific applications made of DAWN; 

■ Local area needs for data on drug abuse trends and consequences; 

■ DAWN’S ability to function as an early warning system and specific design features that 
inhibit, or would facilitate, this application; 

■ Relative sophistication of DAWN’S users in analyzing, interpreting, and explaining the 
data; 

■ Degree to which DAWN suggests trends similar to other data systems; 

■ Value of the current set of oversampled metropolitan areas; 

■ Need to reduce, change, or add items in the data set; 

■ Importance of timely feedback to assist drug enforcement, prevention, and treatment 
efforts, and definitions of “timely” data; 

■ Appropriateness of hospital EDs and ME/Cs offices as data collection sites; 

■ Other strengths and weaknesses of the data system as it is currently designed; and 

■ Alternative data sources available to various constituent groups. 

These topics were discussed with groups known to be current users of DAWN. In semi- 
structured interviews, representatives from DEA, FDA, and ONDCP provided information about 
their uses of the data and suggestions for an alternative design. Second, input was obtained 
from NIDA’s Community Epidemiology Work Group (CEWG) through an internet-based 
discussion forum augmented with telephone conversations. The CEWG relies on DAWN as 
one of several sources of data on drug trends in the Nation’s major metropolitan areas. In 
addition, an extensive review of the published academic literature was conducted to learn more 
about the uses made of DAWN by other drug abuse researchers. 

To gather information from potential users of DAWN, one focus group was conducted in 
each of five different cities. Sharing the perception of the 1997 OAS review panel, it was 
assumed that DAWN’S potential was greatly under-realized at the local level, and that 
community-based researchers outside of the CEWG would be comparatively less familiar with 
DAWN. This broad group - including health care providers, substance abuse treatment 
providers, law enforcement officials, and prevention agencies - was expected to comprise 
“potential” users of DAWN. The purpose of these focus groups was to ascertain their needs for 
data on drug abuse, and the degree to which DAWN could (in its current or future form) fulfill 
those needs. 
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Through this constituent analysis, seven major applications of the DAWN data were 
identified: 

■ Drug scheduling and drug labeling decisions by DEA and FDA; 

■ Indicating the extent of the nation’s drug problem (primarily for ONDCP); 

■ Assessing the effectiveness of local anti-drug efforts; 

■ Guiding resource allocation decisions (for law enforcement, interdiction, treatment, and 
prevention services); 

■ Surveillance of local area drug trends; 

■ Documentation of historical drug trends; and 

■ Source data for academic research on drug abuse. 

By focusing on DAWN’S inability to generate substance abuse prevalence data, or its 
inadequacy in serving as an early warning system, critics of DAWN have overlooked many of 
the potential applications of the system that make it valuable, in many unique ways, to a wide 
variety of constituents. 

With these audiences and their data needs in mind, the strengths of DAWN were then 
identified, and design recommendations were formulated in such a way as to protect and 
improve upon these factors. In addition, weaknesses in the system were also identified, and 
design recommendations considered whether improvements could be effected or certain 
features should be discontinued. Finally, to the extent that DAWN data are used 
inappropriately, recommendations were made for more and better education of DAWN’s users. 
The following sections summarize key design recommendations that are responsive to the 
needs of DAWN’s current and potential users, while recognizing the limitations inherent in any 
single data system. 


How can DAWN be made more useful? 

This was the first major question posed to the redesign team. Given the audiences for 
DAWN, the uses currently made of the system, and interest expressed in new applications for 
DAWN, the ED and ME components were examined closely to identify changes that might be 
made to enhance their utility. Because the two components have substantially different 
audiences and are used for different purposes, each component was examined separately and 
is addressed separately in the sections that follow. 


KEY ISSUES: EMERGENCY DEPARTMENT COMPONENT 

Should DAWN continue? 

The 1997 OAS review panel was convened to address one major question: Should DAWN 
continue? The panel discussed at length the problems with DAWN but also affirmed the value 
of the system. The panel suggested several changes that might make the system more useful 
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and recommended further investigation of local-area applications of the DAWN data. Similarly, 
the redesign team’s constituent analysis found considerable support for DAWN despite much 
criticism. Persons with an ongoing need for information on drug trends made it clear that 
DAWN has substantial potential and should continue. The development of recommendations 
for design changes was mindful of the specific uses that these constituencies will be making of 
the data provided by DAWN. 


Where should DAWN be collecting data? 

The redesign team addressed two general issues about where DAWN should be collecting 
data. The first issue focused on the units of analysis - that is, whether EDs continue to be 
appropriate settings for collection of data on drug-related morbidity, and whether expansion to 
other health care settings is warranted. The second issue focused on the geographic locations 
for which DAWN provides estimates, and whether and how the system’s geographic coverage 
might be expanded. This section briefly discusses each of those issues in turn. Because these 
two issues are critical to the proposed design, more detail is provided about each in separate 
chapters within this report. 


Emergency Departments and Health System Change 

Participants in the 1997 OAS review panel raised questions about the impact of health 
system change on DAWN. Specifically, participants were concerned that changes in managed 
care and other features of the U.S. healthcare system have systematically affected the 
population seen in EDs, thereby compromising the validity of DAWN as an indicator system. 

The major question stemming from this discussion was whether data collection for DAWN 
should be expanded to include other health care settings outside of EDs, such as urgent care 
facilities. 

A systematic review of the research literature, secondary analyses of available data, 
discussions with emergency physicians, and focus groups with medical personnel in four cities 
were conducted. The redesign team found that although the health system has changed 
substantially over the past 30 years, EDs continue to be the most important settings for 
systematic identification of persons whose health is significantly compromised by drugs and 
alcohol. Although recent changes in healthcare delivery have resulted in some “leakage” of less 
urgent drug-related cases from EDs, other changes have resulted in EDs being “flooded” with 
less urgent drug-related cases. The validity of DAWN does not appear to have been 
fundamentally undermined by these changes, but the problem is extremely difficult to assess. 
Physicians argued that expansion of DAWN to urgent care centers, primary care settings, or 
other non-ED facilities would yield few additional cases (and little useful knowledge) relative to 
the increased costs of data collection. Moreover, any expansion beyond EDs would need to be 
tailored to the specific circumstances of each community, as health system change and its 
effects have been highly localized. 

To summarize, the available evidence suggests that the most urgent substance-related 
health problems are still seen in EDs, and this pattern appears to be relatively impervious to the 
effects of health system change. Less urgent substance-related conditions are treated in a 
variety of settings in addition to EDs, but the number and characteristics of these vary across 
communities and over time, making data collection there infeasible and interpretation of data 
problematic. Based on these findings, it was recommended that DAWN continue to collect data 
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on drug-related morbidity from EDs only. Detailed information on the redesign team’s 
assessment of this issue is provided in Chapter 2, Assessing the Impact of Health System 
Change on DAWN. 


Expanding PAWN’S Geographic Coverage 

In 2001, about 470 EDs reported data to DAWN. Data from these facilities are used to 
generate estimates of the number of drug-related ED visits in 21 major metropolitan areas as 
well as the coterminous United States. The redesign team’s constituent analysis indicated that 
both national and subnational (metropolitan area) estimates are desirable and useful. The 
specific metropolitan statistical areas (MSAs) currently included in DAWN are largely an artifact 
of the system’s history, and these areas do not reflect the current distribution of the nation’s 
population. Precision of the DAWN estimates varies considerably across MSAs and the 
national panel, and better and more consistent precision is desired. Precision requirements, in 
turn, drive the estimated sample size requirements. 

After considering many options and their ability to serve DAWN’S current and potential 
audiences, the redesign team recommended that DAWN be expanded to include 48 
oversampled metropolitan areas and to generate reasonably precise estimates for the entire 
nation. Because there is significant demand for metropolitan area estimates in 20 of the 21 
current DAWN MSAs, it was recommended that all of the current areas be retained in the new 
design. Additionally, it was believed that the addition of more metropolitan areas would serve 
the needs of an even broader audience. Thus, the redesign team recommended that DAWN 
include the 5 most populous metropolitan areas in each of the nine Census divisions, plus all 21 
of the current MSAs (see Table 1 -1). This resulted in a total of 48 metropolitan areas. The 
proposed design adds many of the nation’s largest MSAs to DAWN and provides some 
coverage of the less populated census divisions. As an added benefit, increasing the number of 
metropolitan areas means that proportionally fewer of the EDs in the sample will be located in 
the national panel (i.e., that portion of the country outside of the oversampled MSAs). 

The estimated total sample size for the proposed design is approximately 950 EDs. The 
proposed sample design improves significantly the relative standard errors for estimates of total 
DAWN episodes and the major drugs of abuse, which should result in analysts having more 
confidence in the data. At the same time, the proposed sample design permits calculation of 
estimates for the entire nation, not just the coterminous United States, which should help with 
interpretation and comparison of results from DAWN. Specific details on the recommended 
design and factors influencing this recommendation are provided in Chapter 3, Development of 
a New Sample Design for DAWN. 
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Table 1-1. Metropolitan Areas in Proposed Sample Redesign 


Census Division 

Metropolitan Statistical Areas 

NEW ENGLAND 

‘Boston, MA Providence, Rl 

New Haven, CT Springfield, MA 

Hartford, CT 

MIDDLE ATLANTIC 

‘New York, NY Pittsburgh, PA 

‘Philadelphia, PA ‘Newark, NJ 

Nassau-Suffolk, NY ‘Buffalo, NY 

EAST NORTH CENTRAL 

‘Chicago, IL Cincinnati, OH 

‘Detroit, Ml Indianapolis, IN 

Cleveland, OH 

WEST NORTH CENTRAL 

*Minneapolis-St. Paul, MN Omaha, NE 

*St. Louis, MO Wichita, KS 

Kansas City, MO 

SOUTH ATLANTIC 

‘Washington, DC Tampa-St. Petersburg, FL 

‘Atlanta, GA ‘Miami, FL 

‘Baltimore, MD 

EAST SOUTH CENTRAL 

Nashville, TN Knoxville, TN 

Louisville, KY Mobile, AL 

Birmingham, AL 

WEST SOUTH CENTRAL 

Houston, TX San Antonio, TX 

‘Dallas, TX ‘New Orleans, LA 

Fort Worth, TX 

MOUNTAIN 

‘Phoenix, AZ Salt Lake City, UT 

‘Denver, CO Tucson, AZ 

Las Vegas, NV 

PACIFIC 

‘Los Angeles, CA Oakland, CA 

Riverside-San Bernardino, CA ‘Seattle, WA 

‘San Diego, CA ‘San Francisco, CA 

Orange County, CA 


* Area is included in the current design. 


Is DAWN asking the right questions? 

The redesign team also was asked to consider whether DAWN was collecting appropriate 
information about drug-related ED visits, given the uses of the data by key constituent groups. 
Addressing this question called for a reconsideration of the DAWN “case definition” (the criteria 
with which reportable cases are identified) and data elements (the specific information recorded 
about each case on the DAWN report form). It was clear from the constituent analysis and 
review of published literature that the DAWN case definition introduces considerable confusion 
for data analysts, and some of the existing variables lend themselves to misinterpretation as 
well. In addition, it was both unfortunate and ironic that DAWN uses ED records as its data 
source, but collects no information about the medical conditions for which patients seek 
emergency care. This section provides a brief summary of the key recommended changes. 
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Case Definition 


Briefly stated, the current DAWN case definition attempts to include only those ED patients 
who intended to abuse a substance. While there are multiple criteria for determining 
reportability, “intent to abuse” is the most problematic, because ED records rarely include 
explicit documentation of intent. Emergency care professionals are concerned with providing 
expedient treatment of a patient’s injury, illness, or symptoms. Identification and documentation 
of the circumstances surrounding the injury or illness are not high priority activities. DAWN data 
collectors must often infer “intent” when reviewing patient charts. In practice, any use of an illicit 
drug is presumed to be with intent to abuse, and use of prescription and over-the-counter drugs 
is reportable if the ED record indicates suicidal gestures, dependence, recreational use, or use 
contrary to directions. Because case identification requires reporters to make inferences from 
incomplete documentation, the current case definition is applied inconsistently. 

The current case definition also excludes certain classes of drug-related ED visits that are 
of interest to key constituent groups. ED visits related to alcohol (in the absence of other drugs) 
are not currently reportable. Likewise, victims of drug-facilitated assaults are not reportable to 
DAWN, because the victims did not intend to abuse the drug that led to their ED visits. In 
addition, a significant number of individuals are treated each year in EDs for adverse reactions 
to prescription or over-the-counter drugs taken according to direction. For post-market 
surveillance, both FDA and pharmaceutical manufacturers would benefit from including these 
cases in DAWN. 

The redesign team proposed and tested a revised case definition, which sought to “cast a 
broad net” and collect information on a wide variety of drug-related ED visits. That “broad net” 
includes all ED visits for which documentation in the patient’s presenting complaint, 
assessment, or diagnosis indicates that the condition being treated was related to the ingestion 
of a substance. Illicit, prescription, and over-the-counter drugs are included, as are inhaled 
nonpharmaceuticals. The types of cases included and a proposed system for sorting them are 
depicted in Figure 1-1. This approach requires minimal decision-making on the part of 
reporters. This should reduce the learning curve for new reporters and reduce inter-reporter 
variability in applying the definition. The breadth of the definition used in conjunction with new 
data elements will increase end-users’ flexibility in analyzing the data. Finally, the proposed 
expanded definition would continue to capture core drug abuse cases of interest to current 
DAWN users, while also capturing adverse reactions and other toxic effects, allowing DAWN to 
expand its constituent base. 
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Figure 1-1. Type of Case Classification. What type of case is it? Using information 
documented in the patient’s chart, reporters should classify the case as the first type for which it 
meets the specified criteria. 
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Underage Drinking 


Another important change to the case definition is the recommendation to collect 
information on ED visits related to underage drinking. Many of DAWN’S current and potential 
constituents expressed an interest in obtaining data on “alcohol only” cases from DAWN. 
Although the epidemiology of alcohol problems is comparatively well-studied and substantial 
data are available, ED data are not reported by any other system. At present, DAWN is the only 
drug-related national data collection system that explicitly defines alcohol as a non-reportable 
substance (in effect, a non-drug). However, under any definition, alcohol is an illegal substance 
for persons under age 21. 

Interviews with health care providers as part of the redesign team’s health system study 
indicated that among all substance abuse cases, alcohol intoxication cases are most 
susceptible to nonrandom “leakage” from EDs due to local policies for dealing with public 
inebriates. However, youth (under age 21) are not the targets of these policies; therefore, 
collection of “alcohol only” cases for persons under 21 is unlikely to introduce unacceptable 
levels of noise into the DAWN data. 

Field tests in a number of hospitals, along with secondary analyses of data from NEISS, 
suggested that the collection of all alcohol-only cases would substantially increase the volume 
of cases reported to DAWN, perhaps as much as tripling the number of cases reported. An 
across-the-board addition of these cases, along with other proposed redesign changes, would 
exceed OAS’s available resources. However, these same data suggest that the volume of 
underage drinking cases will be minimal but sufficient to address the issue to the satisfaction of 
most constituencies. 


Data Elements 


Since its inception in 1972, DAWN has collected information on drug-related ED visits, but it 
has collected virtually no information about the medical conditions for which patients seek care. 
At the same time, it has attempted to collect some information that ED records are ill-suited to 
provide. The redesign team’s recommendations for changes to the information collected by 
DAWN considered the needs of DAWN’S users as well as the constraints imposed by relying on 
second-hand review of medical charts. Key recommendations are summarized here and in 
Table 1-2. 

DAWN was initially implemented by the DEA, a law enforcement agency. Some of the data 
elements collected in DAWN reflect a law enforcement application of the data. For example, 
DAWN reporters have been asked to determine the form in which the patient acquired the drug 
(not the form in which it was consumed), as well as the source of the drug (e.g., street buy, legal 
prescription, unauthorized procurement). These pieces of information are generally not 
recorded in ED charts, because the information is irrelevant to emergency care staff in their 
diagnosis and treatment of the patient’s presenting complaint. In the final data file, the majority 
of cases are missing this information. In addition, reporters may be tempted to assume the form 
or source of certain drugs and complete the item based on assumption rather than documented 
evidence. Because this information cannot be collected reliably, and because the amount of 
missing information creates problems in interpretation of the data, the redesign team 
recommended that DAWN discontinue collection of these items. 
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DAWN also collects information on route of administration of a drug (e.g., injection, oral, 
inhalation). Like “form” and “source,” this information is not reliably documented. Better 
information on trends in route of administration is available from treatment data sets such as 
Treatment Episode Data Set (TEDS), because drug treatment providers routinely ask about and 
document this information. However, this particular item could be of value to DAWN for 
correctly classifying inhaled substances. The redesign team recommended that 
nonpharmaceuticals be reportable to DAWN only if there is documentation that they were 
inhaled. Collection of limited route of administration data will help analysts ensure that those 
drugs are correctly classified. 

The redesign team also recommended the collection of a number of data elements to better 
describe the medical condition of the patient being treated in the ED. These include some 
information about the patient’s presenting complaint, the nature of the drug involvement in the 
presenting condition, the diagnosis(es), and more detailed information on the patient’s 
discharge status. Finally, collection of a case description through an open-text field was also 
recommended. This information could assist in documenting the circumstances of the case, 
allowing data collection staff to confirm whether it was in fact reportable to DAWN, and possibly 
helping analysts identify emerging patterns in cases involving new drugs or new drug 
combinations. 


Case Identification 


The redesign team was also asked to look at the process by which cases are screened for 
reportability to DAWN. Currently, three methods are used. The preferable method is called 
“100% chart review,” in which reporters attempt to review all of the ED charts to identify DAWN 
cases. As of May 2001, just over 50 percent of the participating hospitals were using this 
method. Other facilities have elected to use a different method, either because the reporter 
cannot get access to all of the charts or because there are so many charts that the reporter 
cannot review them all. About one-third of the participating hospitals use a “log screen” method, 
in which the reporter reviews the presenting complaints recorded in the ED case log to identify 
patients with a high likelihood of being reportable to DAWN. The charts for those likely cases 
are then reviewed. The remaining hospitals use an “ICD-9-CM screen” method, in which a 
hospital staff member provides output from a data file containing the ICD-9-CM codes assigned 
to each visit, and the DAWN reporter reviews those to identify patients with a high likelihood of 
being reportable to DAWN. The charts for only those “likely” cases are then reviewed. 

Although these alternative case identification methods may be helpful in reducing the 
burden on reporters, they have not been systematically assessed. Because each hospital sees 
a somewhat different patient population, and because each facility has different documentation 
and record-keeping styles, the log and ICD-9-CM screens must be adapted to each individual 
hospital. Unfortunately, this means that there may be substantial variability across facilities in 
the sensitivity and specificity of the methods used. It is unknown how many DAWN-reportable 
cases are missed by these screens. 
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Table 1-2. Recommended Data Elements for DAWN (ED Component) 


Data Element 

Recommended change 

Explanation 

Provider number 


A unique hospital identifier, for SAMHSA use only. 

Cross-reference 


A record identifier for reporter use only. 

Date and time of ED Visit 

Month of visit 

For SAMHSA use only. Date/time of visit, when combined with other 
case characteristics, could potentially identify an individual. Month of 
visit is the smallest unit needed to process and report DAWN data. 

Age 

Remove age limits 

DAWN should collect data for patients of all ages. To protect 
individual identities, age categories should be collapsed for all 
patients under age 6 and over age 85. 

Sex 



Ethnicity/race 

White/Black/Hispanic/Other 

Reporters have substantial difficulty with the ethnicity category (not 
understanding it, or the information is not available). DAWN should 
revert to the 4-category combined race/ethnicity variable. 

Patient’s home ZIP Code 

Living arrangements (fixed address, no 
fixed address, not documented) 

ZIP Code data are often unavailable, or require reporters to access 
separate databases. The data are not available for public use, and 
can potentially be used to identify individual patients. Because 
policies toward the homeless can have a substantial effect on ED 
case volume, the redesign team recommended changing this variable 
to focus on whether or not patients are homeless. 

Case description 

Open-text field in which reporters record 
information about the presenting complaint 
and assessment. 

This is a new data element. For SAMHSA use only. 

Reason for taking 
substances 

Do not collect. 

Eliminate from current form. Replace with “Type of case” (see 
below). 

Type of case 

Eight categories, described in flow chart 
above. 

This is a new data element. 

Reason for present contact 

Do not collect. 

Eliminate from current form. Replace with “Presenting complaint” 

(see below). 

Presenting complaint or 
condition 

Categories to identify the type of medical 
condition for which the patient sought 
treatment in the ED. To be developed. 

This is a new data element. An initial list of conditions could be 
specified, but the data collection team should monitor information in 
the “case description” variable over the first several months of data 
collection to develop and refine a comprehensive and analytically 
useful list. 

Diagnosis(es) 

Text field; record all listed diagnoses. 

This is a new data element. Diagnoses could be categorized as 
needed to meet specific research questions. 

Alcohol involved? 

None 

To be retained. 
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Table 1-2. Recommended Data Elements for DAWN (ED Component) (continued) 


Data Element 

Recommended change 

Explanation 

Drug(s) involved - list all 
substances separately 

Allow reporters to list up to 6 substances 
plus alcohol (currently 4 + alcohol). 

Addition of Rx/OTC reactions may necessitate these additional fields. 

Form in which drug was 
acquired 

Do not collect. 


Route of administration 

Change to include only four categories: 
injected, oral, 

inhaled/sniffed/snorted/smoked, 
information not documented. 

For SAMHSA use only. This data element will be used only to 
facilitate correct classification of nonpharmaceuticals (i.e., inhalants), 
as well as to differentiate prescription and over-the-counter medicines 
available in different forms (e.g., antihistamine pills versus nasal 
sprays). 

Source of substance 

Do not collect. 


Disposition 

Revise to include more specific categories: 

(a) Treated and released (discharged 
home, transferred, sent to drug 
treatment/detox, released to police 
custody); 

(b) Admitted (ICU/critical care, 

medical/surgery, psychiatric unit, chemical 

dependency/detox unit); 

(c) Left against medical advice; 

(d) Died; 

(e) Information not documented 

In recent years, about 48% of DAWN cases were “treated and 
released,” while another 48% were “admitted.” More information is 
needed to differentiate patients in these categories. In particular, 
more information is needed to indicate when the ED functions as a 
route of entry into the substance abuse treatment system. 

Coded remarks 

Do not collect. 


Urgency/acuity 
(initially proposed) 

Do not collect. 

The redesign team’s field test indicated that because of substantial 
variation in ED triage and documentation procedures, these data 
cannot be collected using a standard set of categories. 

Insurance coverage 
(initially proposed) 

Do not collect. 

The redesign team’s field test indicated that this information is not 
readily available to DAWN reporters. 



The redesign team was asked to look at the application of the log and ICD-9-CM screens in 
two hospitals. The goal of this limited study was to assess the sensitivity and specificity of each 
screen under the current case definition versus the proposed “broad net” case definition. In 
other words, the redesign team was asked to assess what proportion of reportable cases were 
correctly identified by each screening method, what proportion of non-reportable cases were 
correctly ruled out by each method, and whether the rate of false positives and false negatives 
was substantially different in the current versus proposed case definitions. The concern was 
that broadening the case definition would require adding so many additional presenting 
complaints or diagnoses to the screens that the screens themselves would not be specific 
enough to be useful. The redesign team was not asked to perform a system-wide assessment 
of these screens, nor to assess the variability across hospitals in the application of these 
screens. 

The screening method study yielded three major findings. First, both the log screen and the 
ICD-9-CM screen, using a set of complaints/codes customized for the particular ED, 
successfully identified about 70 percent of the reportable DAWN cases. This means that, 
assuming the study period was representative of the hospital’s usual case flow, reporters in 
either facility would miss about 30 percent of all reportable cases by relying exclusively on the 
screening method. Second, the sensitivity of the screening method was not appreciably 
different between the current case definition and the proposed “broad net” definition. The 
specificity of the log screen became marginally worse with the new case definition (because so 
many more presenting complaints need to be added to the screen), while the specificity of the 
ICD-9-CM screen was marginally better with the new case definition (because ICD-9-CM codes 
exist for the newly-added adverse reactions and drug toxicity cases). In any event, alteration of 
the case definition seems to pose less of a problem than the relatively low underlying sensitivity 
of the screens to drug abuse cases. 

Third, it is unlikely that the true sensitivity of either of these two methods can be reliably 
tested, given the real-world conditions under which medical records are created, used, and 
stored. Even in the 100 percent chart review hospitals, it is unknown how many patient charts 
are moved out of the ED before the reporter can obtain access to them. A fundamental problem 
- and a reality of collecting data in emergency departments - is that different hospitals manage 
medical records differently. Data collection for DAWN must be cognizant of inter-hospital 
variation in medical record-keeping systems. Systematic differences between hospitals must be 
identified and documented. The degree of access to the population of medical records must be 
quantified before any specific data collection strategies are developed. As the new hospital 
sample is implemented, some norms may need to be established so that OAS can make 
informed decisions about the level of medical record access required for a hospital to participate 
in DAWN. Hospitals unable to provide access to the required proportion of charts could be 
replaced with facilities better suited to DAWN participation. Direct review of all available 
medical records must be the gold standard to which DAWN reporters are held. Screening 
methods may lighten reporters’ workloads, but they cannot ensure complete and consistent 
case identification. 


KEY ISSUES: MEDICAL EXAMINER COMPONENT 

Should DAWN-ME continue? 

At the outset of the redesign project, there was some skepticism as to whether the DAWN- 
ME component was sufficiently useful to be worth saving. Little was known about the audience 
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for the ME data, OAS itself lacked confidence in the system, and there was some concern that 
the investment required to repair the system would be prohibitive. The initial constituent 
analysis activities, which focused on the ED component, identified only minimal use of the 
DAWN-ME data. Still, OAS is required by law to collect data on drug-related deaths. The 
redesign team was asked to consider whether this mandate could be effectively fulfilled by the 
use of an alternative data source, and whether there exists sufficient demand to warrant 
continuation of the DAWN-ME component in some revised form. 

A more focused review of the users and uses of drug-related mortality data revealed the 
importance of DAWN as a surveillance system and suggested its unrealized potential. The 
redesign team spoke with a number of participating ME/Cs, exhibited DAWN booths at the 2001 
annual meetings of the American Academy of Forensic Sciences (AAFS) and the National 
Association of Medical Examiners (NAME), and reviewed published analyses of drug-related 
mortality data. They also reviewed other major data collection efforts that provide information 
on drug-related deaths, to see whether any of these might be suitable substitutes for DAWN. 

Key findings from those activities are briefly summarized here. 

The DAWN-ME data are a unique source of information on drug-related mortality in the 
United States. DAWN is the only large-scale surveillance system that collects data directly from 
MEs and coroners, including detailed information on all illicit or abused substances and related 
metabolites detected in the decedent. This differs from other data sources, such as national 
vital statistics data, which list only a few drugs or drug categories and are limited by the short list 
of specific drugs available when using ICD-9 or ICD-10 codes. Moreover, national vital statistics 
data rely on death certificates, which are not reliably updated with information obtained by 
ME/Cs after the completion of a full death investigation (which usually includes toxicology test 
results). 

DAWN is unique in the level of drug detail it collects and reports. In addition, it is the only 
system of its type to provide data for specific metropolitan areas, and it can provide data at the 
jurisdiction level. Moreover, data on drug-related deaths provide an important complement to 
the ED data, as they show another indication of severity or danger associated with certain 
drugs. Given the unique features of DAWN-ME data, it was concluded that with some changes 
to improve DAWN’S timeliness and dissemination methods, new audiences might be reached, 
and new applications might be realized. 

Discussions with ME/Cs revealed substantial interest in the type of information DAWN 
provides, particularly at the jurisdiction level. ME/Cs noted that there is no other data set that 
provides the type of information that DAWN can offer. DAWN provides ME/Cs with information 
on drug-related deaths in “their own backyards” - information that is needed for effective 
surveillance of local drug trends. Moreover, MEs have indicated a readiness for electronic data 
collection, with nearly half of the current DAWN participants submitting data on-line as of July 
2001. Most of the MEs who were contacted by the redesign team viewed participation in 
projects such as DAWN to be part of their professional obligation, and, except for access to their 
data, few other incentives were mentioned as being needed to encourage participation. In fact, 
several MEs approached staff at the DAWN exhibit booths and offered to begin submitting data 
to DAWN immediately, for no payment other than defraying staff costs to process the data. 
Based on input from ME/Cs themselves, it was clear that the DAWN-ME data held great 
potential, which could be met by providing more jurisdiction-specific feedback and eventual 
database-querying capabilities. 
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Access to “real-time” data would provide practical advantages for participating ME/Cs. 
Although identification of new drugs and emerging trends is important for public health 
surveillance purposes, ME/Cs also need this information to more efficiently and effectively 
conduct death investigations. ME/Cs often request standard toxicology panels - that is, they 
have a set list of drugs for which they routinely test. As a result, these standard panels are not 
useful for identifying new drugs of abuse. Because of the expense associated with certain drug 
tests, and the time lag involved when more tests are ordered, ME/Cs generally will not order an 
additional test unless they have reason to suspect a particular drug was involved in the death. 
With timely feedback of data to participating ME/Cs, DAWN can provide information with which 
to assess trends in specific drugs in the ME’s jurisdiction and the surrounding areas - that is, it 
can identify an increase in deaths associated with a given drug. Likewise, DAWN can provide 
basic descriptive characteristics of deaths in which a given drug was involved. With this 
information, ME/Cs can make more informed decisions about when to order toxicology tests for 
drugs that are not included in their standard panels. 


Where should DAWN-ME be collecting data? 

A review of the published literature showed common use of vital statistics data (at the state 
or national level) by researchers interested in obtaining population-based rates of drug mortality, 
or to monitor year-to-year trends in deaths associated with specific drugs. Those researchers 
seemed well aware of the limitations of vital statistics data, and drew appropriate conclusions 
based on available information. Given their interests (population-based rates, national trends, 
“drugs of abuse” as a general analytic category) and their relative analytic sophistication, most 
of these researchers obtain sufficient information from vital statistics data. A redesign of DAWN 
to provide national estimates would largely duplicate existing systems. 

The redesign team’s constituent analysis also discouraged the development of national 
estimates. With the exception of ONDCP, there was little or no support for a redesign of the ME 
component to generate national estimates for DAWN. Rather, there was substantial interest in 
local data (county, jurisdiction and metro area), as well as interest in expanding the number of 
metro areas reporting to DAWN-ME. 

There are also technical limitations to the development of national estimates from DAWN. 
Although not insurmountable, these issues would introduce unknown degrees of bias into the 
resulting estimates. Of particular concern, developing estimates from ME/C data assumes 
consistency in the source of data across jurisdictions. There are more than 2,000 ME/C 
jurisdictions nationwide, and there are substantial differences between states (and among 
jurisdictions within states) in the type of cases accepted for review, the review processes used 
(e.g., toxicology test protocols), and the qualifications/training of staff involved. In addition, 
without polling each jurisdiction, the total number of cases processed by each office each year 
is unknown; such information is crucial to developing a sampling strategy. These issues would 
introduce unavoidable measurement and sampling error into the estimates developed. While 
these problems might be resolved through a lengthy and systematic review of the national death 
investigation system, it remains the case that the key users of data on drug-related mortality 
have not shown sufficient interest in national estimates from DAWN to warrant the effort 
required to produce them. 

Instead, considering user needs and available resources, the redesign team recommended 
that DAWN expand its coverage of ME/C jurisdictions within the current 21 MSAs, prioritizing 
the recruitment of the largest jurisdictions not currently participating. Further, it was 
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recommended that as new metropolitan areas are added to the DAWN-ED component, DAWN- 
ME should seek participation of medical examiners and coroners in those areas as well. At 
present, DAWN receives data from 47 percent of the counties in the 48 proposed metropolitan 
areas. Those participating counties are home to 68 percent of the total population in those 
MSAs. Thus, although about 160 jurisdictions need to be recruited to achieve full participation 
in each MSA, more than two-thirds of the total target population is already covered by DAWN. 
With morbidity and mortality data available for selected metropolitan areas, DAWN can make a 
substantial contribution to a local knowledge about the most serious health outcomes 
associated with drug use. 


Is DAWN-ME asking the right questions? 

The ME component faces a different set of challenges than the ED component in terms of 
the application of the case definition. DAWN reporters in participating ME/C offices have 
access to the full case file for each decedent and can therefore make a determination about 
reportability based on a complete spectrum of information. Unlike ED physicians, ME/Cs must 
make a determination about the underlying cause of death and will indicate when a specific 
contributing factor (such as drug use) is involved. Because ME/Cs must make some 
assumptions in determining cause of death, the DAWN report form asks reporters to indicate 
whether drug involvement in the death was confirmed or presumed. Thus, reporter inference is 
minimized. 

Some ME/C procedures affect the types of cases reported to DAWN. Some of these are 
factors that DAWN cannot change. For example, some ME/Cs do not run toxicology tests for 
marijuana. This may be because the tests are expensive, marijuana use is relatively common 
compared to other drugs, and/or because marijuana overdoses are not common causes of 
death. Therefore, some ME/Cs never report marijuana mentions for any of their cases. DAWN 
is unlikely to be able to change this practice. Other factors might be changed by providing 
additional training to the reporters or by better understanding the reasons for different reporting 
patterns. For example, some jurisdictions report only drug-induced deaths to DAWN, but never 
drug-related deaths. This would suggest that perhaps those reporters misunderstand DAWN’S 
case definition and are reporting only overdose deaths. On the other hand, this may reflect 
different documentation practices of different ME/Cs. The reasons for these variations among 
jurisdictions need to be documented so that the resulting data may be appropriately interpreted. 

The redesign team recommended relatively few changes to the case definition used in the 
DAWN-ME system, although it was proposed that the DAWN-ME case definition be changed to 
be consistent with the proposed ED case definition. Specifically, the new case definition would 
collect all drug-involved deaths (both drug-induced and drug-related), including deaths causally 
or indirectly related to prescription or over-the-counter drugs, whether or not the decedent had 
intentionally abused them. This change will increase the utility of DAWN for other agencies and 
audiences and should reduce the variability in reporting across jurisdictions by minimizing 
reliance on reporter inference regarding the decedent’s intent. As with the ED component, 
alcohol-involved deaths would be reportable for decedents under age 21. Systematic 
differences across jurisdictions in the documentation and processing of deaths should be 
identified and documented, so that appropriate conclusions may be drawn from the resulting 
data. 

Mindful of the desired uses of these data, the proposed changes to the DAWN-ME case 
definition will ensure that DAWN is asking the right questions. Additional changes are needed 
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to ensure that DAWN is disseminating the right answers. Until 2001, the published reports 
presented the DAWN-ME data in substantially the same format as the DAWN-ED reports. 
Unfortunately, by providing reports that look similar, some users assume that similar 
conclusions can be drawn from the ED and ME data. That is, they assume that the system 
totals shown in the DAWN-ME tables represent national estimates, and they assume that 
metropolitan area totals represent MSA estimates. Neither is the case. The published reports 
provide aggregated data, but the participating jurisdictions are not the result of a statistical 
sample, and coverage is inconsistent across metropolitan areas. To minimize misinterpretation 
of the data, OAS redesigned the DAWN-ME reports beginning with the 2001 publication (2000 
data). The new design focuses on jurisdiction and metropolitan area data, prominently shows 
the specific jurisdictions that are and are not participating in DAWN, spotlights population 
centers within metropolitan areas, and eliminates tables that aggregate system-wide data. 

At the DAWN exhibit booths at the AAFS and NAME meetings, ME/Cs expressed 
considerable interest in jurisdiction-level data. Data aggregated to the metropolitan area level 
only introduced questions about how one jurisdiction compared to another within the same 
MSA. The MSA is a concept defined by the Office of Management and Budget (OMB) for 
purposes not relevant to MEs, whose jurisdictions are most often defined by county boundaries. 
ME/Cs will find more value in, and use for, reports that disaggregate data to the jurisdiction 
level. Unlike the ED component, DAWN-ME has the capacity to provide geographically 
meaningful sub-MSA data and should take advantage of this capability. If data are provided at 
the smallest geographic unit of analysis, users can choose how to aggregate data to meet their 
particular needs. 


CROSS-CUTTING ISSUES 

How can the quality and timeliness of DAWN data collection and reporting be improved? 

Data quality and timeliness are issues that cut across both the ED and ME components, 
and nearly all topics considered by the redesign team. This section summarizes key findings 
and recommendations in two main areas: data collection and processing, and dissemination 
methods. 


Data Collection and Processing 


The redesign team was asked to examine various options for electronic data collection in 
DAWN. At present, DAWN-ED data are collected by reporters who examine ED patient charts. 
The reporters identify the patient visits that meet the criteria to be reportable to DAWN, extract 
the information required by DAWN, write this information onto paper report forms, and ship 
these forms to a central office, where the information is keyed into a computer database. Thus, 
data entry and transmittal in DAWN-ED are largely manual processes involving a large number 
of paper forms. As with any manual process, DAWN data collection can be cumbersome, slow, 
and error-prone. 

DAWN-ED reporters will require electronic data collection strategies that allow them 
mobility within their work environments. Reporters often must access different data sources in 
different physical locations and cannot always complete a DAWN report form while sitting in one 
place. In this sense, paper forms are well-suited for some reporters because they can readily 
be carried from place to place. Computerized approaches must be mindful of this need for 
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mobility, as well as the fact that some reporters do not have designated workstations, secure 
storage areas, access to personal computers (or desks on which to put them), or access to 
telephone or Internet connections. 

By comparison, ME reporters pose fewer technology challenges than ED reporters. DAWN 
reporters in participating ME/C offices generally have access to the full case file for each 
decedent and do not have to assemble pieces of information from different data files in 
dispersed locations. The data collection contractor has developed a data collection application 
called eMERS (electronic Medical Examiner Reporting System), which had been deployed in 
about half of the participating ME offices in DAWN by mid-2001. Indications are that most 
reporters in the ME offices have access to a desk, and reporters in the facilities using eMERS 
have access to a desktop computer with an Internet connection. As of 2001, OAS has not yet 
had to supply a computer or an Internet account to any of the eMERS participants. Given the 
current state of data collection in DAWN-ME, the redesign team focused its attentions on the ED 
component. 

Several electronic data collection options for DAWN were considered. These included a 
personal digital assistant (a handheld device), a data tablet (a larger handheld device with a 
keyboard and touchscreen), and a laptop or desktop system. The data collection contractor has 
developed and begun to deploy a functional Internet data collection system for EDs that was 
similar to the ME system; it is known as eHERS (electronic Hospital Emergency Reporting 
System), and had been installed in several hospitals by mid-2001. Because the development 
and deployment of this system occurred simultaneously with the redesign effort, the redesign 
team did not assess this option other than as a point of reference for stand-alone technologies. 

In addition, the redesign team considered the feasibility of near-term implementation of an 
automated electronic data interchange (EDI) process that could gather data directly from 
computerized ED data systems. 

Although palmtop and tablet devices offer maximum portability, it was found that many of 
their hardware, memory, and ergonomic features were extremely limiting. A substantial 
challenge involved storage of DAWN’S detailed drug reference vocabulary, a list that presently 
includes upward of 10,000 drug names. This caused both systems to slow considerably, and 
the long list of drugs made it easy for a user to select an incorrect entry using the stylus. Entry 
of a previously-unrecorded drug was equally cumbersome, and this did not bode well for the 
open-ended case description and diagnosis fields proposed for the revised data collection form. 
Because of problems such as these, further development of the palm and tablet devices was 
discontinued. However, it should be noted that the technology in this field is rapidly evolving. 

As vendors continue to improve the range of software and hardware options for handheld data 
recording devices, it may become more likely that a specific program can be adapted for 
DAWN. 

An immediate solution to the portability issues that are central to ED reporters is the 
development of data entry software for a laptop computer. Laptops and desktops offer larger 
screens, permitting a preferable layout of the electronic form. Likewise, personal computers 
have little difficulty searching the lengthy drug list, and full-size keyboards allow for easy text 
entry. Data can be securely uploaded via modem to a central server. An application developed 
by the redesign team showed considerable promise but for various reasons could not be 
subjected to a rigorous field test during the performance period of the contract. Such a test is 
required before implementing computerized data collection in DAWN. Additionally, screen 
layouts for the final laptop/desktop application should be substantially similar to the Internet 
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application, facilitating the development of a single set of training manuals, user guides, and 
helpdesk functions. 

Any computerized data entry system for DAWN must permit reporters to submit data on a 
flow basis. This is not presently the case. Instead, data are captured and batched only after the 
end of a predefined reporting period (typically a 7- to 10-day period), a holdover from historical 
paper data collection procedures. In other words, the current paper and electronic systems 
impose constraints on the frequency with which reporters can physically submit data. As other 
redesign activities have shown, real-time submission, analysis, and dissemination of DAWN 
data are essential to the system’s ability to provide timely surveillance of emerging drug trends. 


Dissemination Methods 


As its name implies, DAWN was designed to provide early warning of emerging drug trends 
at the local and national level. The redesign team provided recommendations for statistical 
approaches and dissemination strategies to accomplish these objectives. They proposed the 
use of statistical process control (SPC) models to identify meaningful deviations from historical 
reporting patterns at both the facility and metropolitan area levels. SPC models can identify, but 
cannot explain, statistically significant deviations in reporting patterns for a given drug, group of 
drugs, diagnosis, or other condition. Investigation of those deviations may reveal data 
processing or analysis errors, important changes in a hospital’s ED population, local health 
system changes, or other factors that might result in unusually high numbers of drug-related ED 
visits. Thus, an SPC approach can simultaneously serve as a quality control process, provide 
historical context for current data, and alert OAS to potentially important shifts in drug trends 
both locally and nationally. 

The potential value of information provided by SPC models highlights the importance of 
facility feedback and broader dissemination of DAWN data. Successful implementation of the 
redesigned DAWN requires a comprehensive information dissemination and constituent-building 
strategy. This strategy must include timely feedback of data to participating facilities, whether in 
hard copy or online. Feedback reports might also highlight missing data to help encourage 
more complete or timely reporting. Feedback reports should include not only frequencies of 
substances reported to DAWN, but some historical and regional comparisons to help 
emergency care staff and ME interpret the data. Efforts must be made to identify and target the 
facility staff who can best make use of the information provided by DAWN. 

The redesign - specifically, the implementation of a new case definition and data elements, 
and the eventual addition of new metropolitan areas - will require changes to the mid-year and 
year-end DAWN reports. In addition to implementing necessary changes to the published 
tables, some revisions could be made to the layout of the reports, with the goal of enhancing 
their appeal and utility for various end-users. A revised ME report is being developed for 
publication of the 2000 DAWN-ME data, and lessons learned from that process may help OAS 
identify important design features for the ED reports. Continued production of The DAWN 
Report will provide opportunities to disseminate brief, focused analyses of special topics. 

The redesign team also recommended dissemination of information about the redesign 
itself, including the reasons for, and implications of, the changes being made to DAWN. Finally, 
the team recommended that OAS establish a presence for DAWN at the meetings of major 
trade associations and professional organizations, such as NAME, the American College of 
Emergency Physicians, the Emergency Nurses Association, and the American Public Health 
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Association. Activities could include exhibit booths, as well as presentations of key findings. 
Panel presentations on DAWN and its potential applications may help publicize the redesign, 
while periodic presentations of research findings can disseminate findings from DAWN while 
modeling appropriate uses of the data. Publications of research findings in peer-reviewed 
journals will further help build and strengthen DAWN’S reputation among the research 
community. 


CONCLUSION 

In an effort that spanned several years, OAS and the redesign team documented the 
primary users of DAWN data, the potential users of a redesigned system, and the major 
applications that are made of the data. Recommendations were made for changes that are 
needed to ensure that DAWN is asking the most appropriate questions, providing clear and 
correct answers, and reaching the most important audiences. All design recommendations 
have been mindful of the way in which the data are used and of the limitations inherent in 
gathering data from a secondary review of medical records and case files. The goal was to 
suggest design features that will preserve the utility of DAWN for its current users, expand its 
reach to new audiences, enhance users’ understanding of DAWN, and improve the quality and 
timeliness of the data. Implementation of a new design represents a critical step in DAWN’S 
history, and provides numerous opportunities for OAS to realize the system’s full potential. 
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2. ASSESSING THE IMPACT OF HEALTH 
SYSTEM CHANGE ON DAWN 


T he Drug Abuse Warning Network (DAWN) monitors the number of drug-related admissions 
to emergency departments (EDs) only. Consequently, DAWN cannot be used to estimate 
drug abuse prevalence but can be used as one indicator of the intensity of the nation’s 
substance abuse problem. DAWN is sensitive to a number of drug-related factors that contribute 
to the intensity of the substance abuse problem in the general population: the price and purity 
of drugs, methods of administration of a drug, dose and frequency of drug use, potency of drug 
combinations, and aging of the substance abuse population. It is expected that increases or 
decreases in these factors will result in corresponding changes in the number of DAWN ED 
episodes and mentions. 

The DAWN redesign team was asked to investigate whether DAWN might also be sensitive 
to health system changes unrelated to substance abuse. Specifically, the 1997 Office of 
Applied Studies (OAS) review panel was concerned that evolving managed care practice could 
result in a shift in the locus of care for substance abuse away from EDs to urgent care and 
ambulatory care centers. A noticeable decline in ED substance abuse cases would make the 
interpretation of DAWN trends and patterns problematic. If health system changes significantly 
altered the composition of the ED patient population, then the number of cases detected by 
DAWN could be unrelated to underlying changes in the intensity of the drug problem. For 
example, increases in the number of medically uninsured could shift utilization from primary 
care clinics to EDs, “flooding” the ED with a poor, uninsured population; managed care practices 
could divert ED utilization to urgent care or primary care settings. 

This chapter explores the scope and size of the effects of health system change on the ED 
population and DAWN counts of substance abuse cases. A literature review identifies some 
factors that can affect the characteristics of this population. Analyses of national data sets 
provide additional information about ED utilization, insurance status, and limitations associated 
with existing data. A qualitative study undertaken to identify local patterns in seeking help for 
medical conditions related to substance abuse is described. Finally, the implications for DAWN 
are discussed. 


REVIEW OF LITERATURE 

There is little empirical research that explains whether and how health care system 
changes over the past 3 decades have systematically affected the substance abuse population 
seen in the ED. Some articles attempt to characterize changes in the ED population over time. 
These articles are largely based on experience in single, urban EDs usually associated with an 
academic institution and are not generalizable to the nation as a whole. Other recent research 
investigates the effects of managed care on ED utilization. These two categories of articles 
describe four major elements of health system change that can impact ED-utilization patterns: 

■ Increases in the numbers of persons who are medically indigent; 
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■ Lack of a primary care provider; 

■ Alternative sites of emergency and urgent care; and 

■ Growth of managed care. 


Increases in the Number of Persons Who Are Medically Indigent 

Changes in the number of medically indigent and homeless persons could increase the 
number of persons seeking care in EDs. The number of uninsured patients is increasing 
nationally. Recent estimates from the Agency for Healthcare Research and Quality (AHRQ), 
based on the Medical Expenditure Panel Survey (MEPS), indicate that 16.8 percent of the U.S. 
population had no health insurance in 1996 (Vistnes & Zuvekas, 1999), up from 12.9 percent in 
1987 (U.S. Census Bureau, 1999). The ED is the solution of last resort for persons unable to 
pay for medical care (Millmann, 1993). The Emergency Medical Treatment and Active Labor 
Act (EMTALA) stipulates that all patients who present to the hospital ED should be evaluated 
and medically stabilized. Unlike other care sites, which may turn away persons for lack of 
funds, hospital EDs are mandated to treat at least minimally all persons presenting for care, 
regardless of the severity of the condition or the ability to pay. 

In many communities, the ED may represent the only source of medical care for persons 
who are medically indigent or homeless. Several sources indicate that the homeless are more 
likely to use the ED than clinics or physician offices as a primary source of medical care (Little & 
Watson, 1996; Malone, 1998; O'Toole, Gibbon, Hanusa, & Fine, 1999; Rask, Williams, Parker, 

& McNagny, 1994). The ED provides important ancillary social services for this population 
(Malone, 1998). Thus, while the ED treats emergency cases that are of particular interest to 
public health surveillance, it also provides non-urgent care for medically uninsured persons and 
other populations without other access to primary care. 

There is some evidence that the medically indigent and the homeless are more likely to 
have substance abuse problems. Survey data from the 1997 National Household Survey on 
Drug Abuse showed that household residents without health insurance were more than twice as 
likely to report past-year illicit drug use than were those who had some form of insurance 
coverage (Office of Applied Studies, 1999a). According to 1997 National Hospital Ambulatory 
Medical Core Survey (NHAMCS) data, ED visits for alcohol- and drug-related problems were 
more than twice as likely to be self-pay as visits for all other problems (NHAMCS public-use 
files). In a study of homeless adults, two-thirds of a sample of 1,563 homeless adults met 
criteria for chronic substance dependence (Koegel, Sullivan, Burnam, Morton, & Wenzel, 1999). 

The data indicate that as more people are without health insurance, persons with substance 
abuse problems are more likely to be uninsured, and the number of uninsured cases treated in 
the ED for substance abuse will increase. These increases reflect a shift of medical utilization 
from primary care clinics to the ED among persons who lose their insurance. This trend could 
increase the number of substance abuse cases treated in the ED, in addition to any increases 
resulting from a larger number of cases in the community or the hazardous characteristics of 
drug(s) in use. On the other hand, fewer patients might be treated in the ED if primary care 
sites treating the uninsured are available. 
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Lack of a Primary Care Provider 


Health system changes affecting access to primary care could affect ED utilization. A 
substantial number of patients use the ED as a source of primary care for the treatment of non¬ 
urgent medical conditions. African Americans and Hispanics are more likely to identify the ED 
as their regular source of health care (Baker, Stevens, & Brook, 1994), as are patients living in 
homeless shelters (Rask et al., 1994). Patients who identify the ED as their regular source of 
care are three times more likely to have used the ED more than once in the preceding year 
(Diamant, Brook, McGuigan, Fink, & Gelberg, 1998). Primary care physicians are less prevalent 
in poor urban neighborhoods and the urban poor lack access to primary care due to lack of 
transportation (Baker, Stevens, & Brook, 1991; Young, Wagner, Kellermann, Ellis, & Bouley, 
1996). 

Substance abusers have been identified as a group that is more likely to use the ED as a 
source of primary care. A study of 500 current chronic drug users in Florida revealed that one- 
third used a community clinic for regular health care, one-third had no usual source of care at 
all, and the remaining third used a hospital ED (McCoy et al., 1997). Substance abusers who 
have not been treated for their substance abuse disorder tend to be high utilizers of medical 
care, particularly of emergency and inpatient services. A substance abuse diagnosis 
significantly increases the risk for repeated ED use (Spooren, De Bacquer, Van Heeringen, & 
Jannes, 1997). Between 40 percent and 58 percent of patients making more than 10 visits to 
an urban ED in a year had psychiatric or substance abuse problems (Kne, Young, & Spillane, 
1998; Rask, Williams, McNagny, Parker, and Baker, 1998; Spillane et al., 1997). Chronic drug 
users were about 30 percent more likely to seek care in EDs than either casual drug users or 
non-users (McGeary & French, 2000). 

There is strong evidence that homeless and minority substance abusers, in particular, have 
traditionally used the ED as a source of primary care. Changes in the medical system that 
systematically redirect patients from ED to other primary care settings could have substantial 
impact on DAWN’S representativeness and related analysis. 


Development of Alternative Sites of Emergency and Urgent Care 

Over the past 2 decades, specialized centers of care have been developed within hospitals 
to expedite services and reduce costs. Ongoing pressure to contain these costs and create 
efficient lines of service will likely expand these specialized services (Johnson, 1998). The 
proliferation of such services could divert patients away from the ED. 

Some of the available specialized services include the following: 

■ Pediatric EDs, specializing in emergency services for children; 

■ Chest pain centers, specializing in diagnosing and treating chest pain related to cardiac 
events; 

■ Psychiatric EDs, specializing in emergency services for psychiatric disorders; 
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Urgent care centers, specializing in expeditious treatment of urgent medical conditions 
that would be assigned a low priority status in an ED; and 


■ Observation units, designed to provide short-term patient beds for conditions that do 
not warrant hospitalization under managed care rules. 

The DAWN data collection contractor has found that 33 percent of participating hospitals in 
the 21 oversampled metropolitan areas have one or more specialty units within the ED - 14 
percent have psychiatric emergency units, 4 percent have emergency surgery units, 14 percent 
have pediatric emergency units, and 9 percent have trauma centers. In some EDs, patients 
admitted to these specialty units are first triaged though the ED admitting process. In other 
EDs, patients are admitted directly to the units. The admission and triage process is important 
for DAWN in that different processes may render different sets of medical records available for 
DAWN reporters. If cases are triaged before they enter the ED, those cases may not be 
available for review and fewer DAWN cases may result. On the other hand, if cases are triaged 
to specialty units after first arriving in the ED, those files may be available for review even 
though the patients were not treated in the ED. 

One particular concern raised by the 1997 OAS review panel was the potential diversion of 
DAWN-reportable cases to free-standing urgent care centers and the need to consider 
incorporating these facilities into the DAWN sampling frame. A variety of urgent care centers 
now operate throughout the United States. Emergi-centers, for example, provide a full spectrum 
of emergency services on a 24/7 basis. Others provide less comprehensive services on a more 
restrictive basis. These centers offer both walk-in and appointment services with a primary care 
provider, and many have expanded hours and shorter waiting times than in EDs (Mezey, 1999). 
Urgent care centers present an additional complication, in that there is wide variability in the 
number of centers serving any location. 

The structure of urgent care centers can vary considerably. Some urgent care centers are 
part of physician group practices with expanded hours. Others are clinics affiliated with 
hospitals, either at the hospital site or in a free-standing location. There appear to be two types 
of patients who find these care centers attractive (Mezey, 1999). The first type wants the care 
and access provided by EDs, without the wait or “negative feedback” for using EDs for non¬ 
urgent problems. The second type includes those patients whose insurance plans treat urgent 
care centers preferentially compared with private physicians’ offices. 

It is possible that freestanding urgent care and emergi-centers treat a substantial number of 
persons with serious drug-related health conditions. There is no literature, however, that 
describes the number of patients treated in these settings for drug-related problems. 


Growth of Managed Care 

Managed care has been one of the most obvious sources of change in the health care 
system over the last 20 years. The 1997 OAS review panel discussed at length the issue of 
managed care’s potential impact on DAWN. The panel assumed that managed care 
significantly influenced the size and composition of the population that uses ED services, 
thereby affecting DAWN’S validity. 
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Managed care is a broad term for practices designed to decrease costs or limit 
unnecessary services while increasing the quality of health care. The literature reveals its 
highly controversial nature. Critics claim that managed care practices decrease short-term 
expenses at the cost of patients’ well-being, creating additional costly medical problems. 
Proponents of managed care point to improvements in monitoring the quality of care, providing 
continuity of care across different sections of the health care system, and curtailing costs that 
would otherwise place medical care outside the reach of many persons. There is little 
consensus about the effects of managed care on utilization and quality of care, in general, and 
little about the effects of managed care on ED utilization, in particular. 

The proportion of persons covered by commercial managed care contracts has increased 
several-fold in the last 15 years. This change affects EDs because managed care plans, and 
especially health maintenance organizations (HMOs), have sought to reduce costs by diverting 
what are viewed as “medically inappropriate,” nonurgent visits from the ED to primary care 
networks or urgent care centers. This should have systematically reduced the number of cases 
seen in EDs. 

Managed care practices discourage persons with nonurgent conditions from seeking care 
at the ED, based on the assumption that nonurgent conditions can be more effectively treated in 
primary care settings and at a much lower cost. The literature describes several managed care 
strategies: preauthorization, denial of payment, special triage telephone numbers, patient 
education, increase in co-payments, primary care outreach, urgent care centers, and risk¬ 
sharing with primary care networks. Preauthorization and denial of payment for treatment of 
non-emergent conditions have been the most controversial—the diagnosis cannot always be 
determined until after tests have been performed at the ED. In order to curtail denial of 
payment, states have enacted a “prudent layperson” standard. This requires the HMO to pay 
for services if the symptoms would be judged by a prudent layperson to be potentially serious. 

Literature that examines the effects of managed care on ED utilization is contradictory. For 
example, studies have documented mixed effects of preauthorization (Fisher, 1999; Hurley, 
Freund, & Taylor, 1989; Ling, Cooke, & Kornfeld, 1995). Researchers at the Center for 
Studying Health System Change have found no evidence to suggest significant effects of 
managed care on ED utilization (Tu et al., 1999). Using private sector data from their 
Community Tracking Study’s household telephone survey, they focused on whether belonging 
to an HMO influenced the report of ED utilization among individuals with private health 
insurance. The model used was careful to correct for local market effects. No relationship was 
found between HMO membership and self-reported, past-year ED utilization. 

Anderson, Zhang and Worzala (1999) examined the effects of low, medium, high, and very 
high HMO market penetration on ED utilization in 298 metropolitan areas over a 12-year period. 
Their evidence points to strong effects of managed care on ED utilization in some areas. In 
addition to a direct effect, HMO penetration had an indirect effect on medical utilization because 
insurers anticipated the coming of HMOs into the local market and behaved accordingly. 

Finally, a 1997 survey of chairs of academic departments of emergency medicine revealed a 
mixed message - 34 percent believed that managed care had been significantly or primarily 
responsible for a low rate of increase in patient volume, while 40 percent believed that managed 
care was not responsible (Counselman et al., 1998). 
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There appear to be several reasons for these mixed findings regarding the effects of 
managed care on ED utilization. First, managed care effects are highly localized. The rate of 
managed care penetration varies significantly by state, by sector, and by metropolitan area. For 
example, in 2000, 90 percent of Medicaid patients were served under managed care contracts 
in Colorado but only 6.3 percent in Louisiana (Health Care Financing Administration, 2000). 
Furthermore, managed care services are more heavily concentrated in metropolitan areas than 
in rural areas (Ricketts, Slifkin, and Johnson, 1995). For example, Sacramento County had 
commercial penetration rates of 75 percent in 2001, even though the California commercial rate 
overall was around 50 percent (Cattaneo and Stroud, 2001). Other factors affecting managed 
care practices include the strength of local provider organizations, the nature of contracted 
benefits, local market competition, and state legislation and regulation. These factors vary 
greatly and change rapidly (Miller and Luft, 1994). A study conducted in one location may 
produce findings contrary to a study conducted in a different location. 

Second, managed care itself is evolving - its effects are difficult to identify and interpret. 

For example, in addition to FiMOs, managed care organizations now offer less restrictive 
Preferred Provider Organizations (PPO) and Point of Service (POS) plans. Though once 
credited with decreasing ED utilization, managed care in the form of less restrictive PPO and 
POS plans may be responsible for recent increases in ED utilization. For example, a Boston 
staff-model PIMO recently transformed itself into a medical group serving multiple managed care 
plans. Under the new structure, reducing ED utilization was no longer financially rewarded, and 
the group scaled back its urgent care network, increasing pressure on local EDs (Brewster, 
Rudell, and Lesser, 2001). 

Third, national statistics about persons served in the ED conceal substantial variation 
across facilities and geographic areas. Even hospitals within the same community can 
experience divergent conditions (U.S. General Accounting Office, 1993). For example, 82 
percent of patients presenting to a Los Angeles public ED in 1992 reported having no form of 
public or private medical insurance (Baker et al., 1994). In contrast, NPIAMCS data for the 
same year estimated that self-pay was the expected source of payment for 14 percent of all ED 
patients nationally. Similarly, the prevalence of substance abuse among ED cases may vary 
considerably among sites. Such variation ranges from 9 percent to 47 percent, depending on 
the patient population, its location, and when the measurement takes place (el-Guebaly, 
Armstrong, & Flodgins, 1998). The correlates of drug abuse also seem to vary. For example, 
alcohol’s involvement in injury varies according to the type and location of ED and the drinking 
patterns of the region in which the ED is located (Cherpitel, 1997). This local variation in 
demographics and substance abuse patterns makes overall change difficult to detect. 

In summary, a literature review yielded little useful information about how health system 
change has affected ED utilization and DAWN counts of substance abuse cases. The review 
did reveal important information about factors that influence where persons seek health care. 
Drug-related conditions and emergent medical illnesses (e.g., cardiac problems) generally result 
in ED utilization. For non-emergent conditions, the decision to access the ED may be mediated 
by lack of insurance, access to other health care settings (including primary care, urgent care, 
or specialty centers), and incentives or disincentives related to managed care practices. 

Because the ED is the only health care setting that is required to treat persons without 
health insurance, lack of medical insurance is a strong predictor of ED utilization for uninsured 
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persons with nonurgent problems. The literature suggests that persons living in neighborhoods 
without easy access to primary care facilities tend to use the ED. Alternatively, persons with 
insurance coverage may find it more convenient to access urgent care and other specialty sites 
rather than the ED. Finally, managed care organizations can divert utilization from the ED 
through a number of different strategies, including higher copayment, preauthorization, and 
refusal to pay for nonurgent visits. 


ANALYSES OF NATIONAL DATA 

This section describes a series of secondary analyses of existing national data that were 
undertaken to identify any systematic changes in ED utilization over the past several years. 

Data from NHAMCS and MEPS were used in these analyses. 

NHAMCS is the most comprehensive source of information on ED utilization in the United 
States. This survey has been conducted annually since 1992 by the National Center for Health 
Statistics (NCHS). Data are collected on patients’ symptoms and demographic characteristics, 
diagnoses, services provided, drugs prescribed, and referral status from a national sample of 
524 non-federal, short-stay general hospitals. Each participating hospital provides encounter 
information on a target sample of 50 ED visits during a 4-week reporting period. About 71,000 
visits are sampled annually. Data are weighted to produce annual national estimates. This 
section describes analyses of NHAMCS public-use data files from 1992-97 (the most recent 
data files available at the time this task was implemented). Unfortunately, there are no known 
analogous data systems providing utilization data in the ED for the years prior to 1992. 

NHAMCS allows for some exploration of the extent of drug-related diagnoses in the ED 
population. NHAMCS originally included a variable on its data collection form noting whether 
the case was alcohol- or drug-related, but it contained a considerable amount of missing data in 
later years. This variable was dropped from the form in 1996. Consequently, analyses in this 
chapter focus on a set of alcohol and drug-related (ADR) ICD-9-CM diagnostic codes. ADR 
cases were defined as any case to which a set of six specific ICD-9-CM codes had been 
assigned: 

■ 291.0 and subcodes (alcoholic psychoses); 

■ 292.0 and subcodes (drug psychoses); 

■ 303.0 and subcodes (alcohol dependence syndrome); 

■ 304.0 and subcodes (drug dependence); 

■ 305.0 (nondependent abuse of alcohol); and 

■ 305.2 - 305.9 (nondependent abuse of drugs). 

These codes identified the most likely ADR cases; no attempt was made to replicate the 
DAWN case definition in the analyses. 
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MEPS is a nationally representative survey of health care use, expenditures, sources of 
payment, and insurance coverage for the U.S. civilian, noninstitutionalized population. It also 
includes a national survey of nursing homes and their residents. This survey is designed to 
yield comprehensive data that estimate the level and distribution of health care use and 
expenditures, monitor the dynamics of the health care delivery and insurance systems, and 
assess health care policy implications. The third of a series of surveys on this topic, MEPS 
follows on the National Medical Care Expenditure Survey (NMCES, also known as NMES-1), 
conducted in 1977, and the National Medical Expenditure Survey (NMES-2), conducted in 1987. 
The MEPS Household survey employs an overlapping panel design to collect medical 
expenditure data at both the person and household levels. These data are then linked with 
additional survey data collected from the respondents’ medical providers, employers, and 
insurance providers. The following analyses draw upon the 1987 NMES and the 1996 MEPS 
public-use data files (Agency for Health Care Policy and Research [AHCPR], 1991,1998). 


National Trends in ED Utilization 

The rate of ED visits remained stable at about 36 visits per 100 persons from 1992 to 1998. 
The overall number of visits to EDs increased from 89.8 million in 1992 to 100.4 million in 1998, 
in proportion to the rise in population (McCaig, 2000; Schappert, 1997). At both points in time, 
the most frequent users of the nation’s EDs were females between the ages of 25 to 44 (roughly 
16 percent of all ED visits), followed by males of the same age range (14 percent of all visits). 

In both 1992 and 1998, whites accounted for about 80 percent of all ED visits, or about 34 visits 
per 100 persons. Blacks accessed the ED at a much higher rate than whites, and this rate 
increased, rising from 55 visits per 100 persons in 1992 to 62 visits per 100 persons in 1998. 
Studies have shown that higher ED rates for minorities may be related to inadequate access to 
primary health care among these populations (Baker et al., 1994; Young et al., 1996). 

Table 2-1 provides some general information about how ED utilization has changed in 
recent years. This table shows weighted estimates from NHAMCS for 1993-97, including the 
total number of ED visits annually and the number of ADR visits. As shown, there was no 
significant change in the number or population-based rate of ED visits from 1993-97. However, 
the proportion of ED visits with ADR diagnoses experienced a small, steady, and statistically 
significant increase from 1993-97. Data from DAWN (not shown) indicate a significant increase 
in the number of DAWN cases over the same 5-year period (SAMHSA 1996a, SAMHSA 1996b; 
SAMHSA 1999b, SAMHSA 1999c, SAMHSA 1999d). 
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Table 2-1. Trends in E 

D Utilization, 

1993-97 (NHAMCS data) 




1993 

1994 

1995 

1996 

1997 

US Population 
(in 1,000s) 

257,746 

260,289 

262,765 

265,190 

267,744 

All ED Visits 
(SE) 

90,265,163 

(4,839,453) 

93,401,037 

96,543,579 

90,345,526 

94,934,191 

(4,553,200) 

Per 1,000 Population 
(SE) 

350.21 

(18.8) 

358.84 

367.41 

340.68 

354.57 

(17.0) 

ADR Visits 
(SE) 

1,160,202 

(97,450) 

1,147,700 

1,376,870 

1,434,310 

1,449,645 

(120,974) 

Per 1,000 Population 
(SE) 

4.50 

(.38) 

4.41 

5.24 

5.41 

5.41 

(.45) 

As % of all ED Visits 
(SE) 

1.30% 
(.08%) 

1.23% 

1.43% 

1.59% 

1.53% 
(.10%) 


SOURCE: analysis of NHAMCS public-use data files, informed by Nourja (1999) and Schappert (1997). 


In order to investigate whether increases in DAWN counts and ADR visits to the ED are 
related to health system change, the following sections examine data on lack of health 
insurance and managed care. 


ED Utilization Among the Uninsured 

Information on expected source of payment in the ED - especially payment information 
specific to substance abuse cases - is difficult to find and interpret. In an effort to keep up with 
rapid changes in managed care and insurance coverage arrangements, NHAMCS changed its 
payment source categories several times between 1992 and 1997. This renders comparisons 
of different timepoints difficult, and it also highlights the problems of trying to collect consistent 
data on insurance coverage, which is constantly in flux. 

NHAMCS does not include an uninsured category on its encounter form. Instead, uninsured 
cases were identified as those with the patient as the expected source of payment (i.e., “patient 
paid”). Because multiple-response categories were permitted in 1993, any visits that may have 
involved copayments (i.e., those that also indicated some form of insurance coverage) were 
excluded from this group. In 1997, the payment variable was structured such that copayments 
were not counted in the “patient paid” category. Table 2-2 compares the proportion of “patient 
paid” cases in 1993 and 1997. These data show that the total number of self-paying patients in 
the ED increased by about 30 percent from 1993 to 1997, and that self-pay visits as a 
percentage of all ED visits rose by about 3 percent (a statistically significant change). More 
importantly, the total number of self-paying patients seen in the ED for alcohol and drug-related 
diagnoses increased by 80 percent, to about 12 percent of all ADR ED visits. In 1997, over 40 
percent of patients seen in the ED for ADR diagnoses had no public or private health care 
coverage for their ED visit. 
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Table 2-2. ED Cases with Patient as Expected Source of Payment, 1993 and 1997 



1993 

1997 

Total Number of ED Visits 

90,265,163 

94,934,191 

ED Visits with Patient as 
Expected Source of Payment 

11,690,000 

15,260,000 

(SE) 

(665,438) 

(789,682) 

As Percentage of All ED Visits 

13.0% 

16.1% 

(SE) 

(.25%) 

(.31%) 

Number of ADR Visits 
to the ED 

1,160,202 

1,449,645 

ADR ED Visits with Patient as 
Expected Source of Payment 

328,324 

593,010 

(SE) 

(43,786) 

(69,677) 

As % of All ADR ED Visits 

28.3% 

40.9% 

(SE) 

(2.9%) 

(3.3%) 


SOURCE: analysis of NHAMCS public-use data files 


These findings may indicate an increase in the number of uninsured persons with 
nonurgent conditions who have accessed the ED over time. If the lack of insurance leads to 
use of the ED rather than other medical sites, DAWN would detect an increased rate of 
substance abuse. An influx of the uninsured into the ED, however, would not result in a 
corresponding national increase in substance abuse problems. In order to understand whether 
such a bias affects DAWN, more information is needed on sources of payment and the acuity of 
DAWN cases. 


ED Utilization and Urgency of Visits 

While no information is available on the acuity of DAWN cases specifically, NHAMCS 
collects this information for its sample. Table 2-3 shows trends in the acuity of ED visits from 
1993 to 1996. In NHAMCS, an urgent visit is defined as involving immediate attention for an 
acute illness or injury that threatens life or function, where delay would be harmful to the patient. 
Nonurgent visits were defined as not requiring attention immediately or for a few hours. Data 
from 1997 were not included in these analyses because changes were made to the acuity 
variable, rendering direct comparisons with earlier years impossible. 

Table 2-3 shows a small (1.2%) but statistically significant increase in the percentage of ED 
visits made for urgent/emergent conditions between 1993 and 1996. There may be several 
reasons for this increase. If managed care has successfully implemented practices that divert 
nonemergent cases from the ED, the proportion of urgent/emergent visits to the ED should 
increase. Alternatively, shortened hospital stays and longer waits for patients attempting to 
access primary care through managed care might lead to the development of more acute 
conditions requiring immediate treatment in the ED. 

Table 2-3 also shows that ADR visits are more likely to be categorized as urgent than are 
visits for other diagnoses. For example, 57 percent of all ADR cases were characterized as 
urgent in 1996, compared to 46 percent of ED visits overall. Although the proportion of all ED 
visits designated as urgent/emergent increased, the increase is in non-ADR visits, which 
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showed no change in urgency. The finding could reflect a relatively low level of statistical power 
due to the small number of ADR visits relative to the extraneous variance in the analysis. 
Alternatively, because a high proportion of ADR visits is already urgent/emergent, ADR visits 
may be relatively impervious to diversionary strategies. In order to assess how vulnerable 
DAWN counts are to strategies that divert nonemergent cases from the ED, more information is 
needed about how many DAWN cases are urgent/emergent. 


Table 2-3. Trends in Urgent/Emergent ED Visits, 1993-1996 



1993 

1994 

1995 

1996 

All ED visits 

90,265,163 

93,401,037 

96,543,579 

90,345,526 

Urgent/Emergent Visits 
(SE) 

40,575,472 

(2,200,483) 

44,090,728 

44,193,451 

41,732,823 

(1,738,503) 

Per 1,000 Population 
(SE) 

157.4 

(8.5) 

169.39 

168.19 

157.4 

(6.55) 

As % of all ED Visits 
(SE) 

44.95% 

(.37%) 

47.21% 

45.78% 

46.19% 

(.42%) 

Urgent+Emergent 
ADR Visits 
(SE) 

722,706 

(70,879) 

663,067 

784,410 

817,346 

(79,091) 

Per 1,000 Population 
(SE) 

2.80 

(.28) 

2.55 

2.99 

3.08 

(.57) 

As % of all ADR Visits 
(SE) 

62.3% 

(3.1%) 

57.77% 

56.97% 

57.0% 

(3.2%) 


SOURCE: analysis of NHAMCS public-use data files 


Effects of Managed Care on Ambulatory Utilization Patterns 

The effects of managed care on ED utilization can be examined by looking at the pattern of 
ambulatory care utilization for both managed care and fee-for-service (FFS) patients. There are 
very few data sources that report data from across the nation’s ambulatory care system. 
Supplemented with data from a companion survey, the National Ambulatory Medical Care 
Survey (NAMCS) serves this purpose well. Table 2-4 shows combined data from NFIAMCS and 
NAMCS describing national utilization of ambulatory services in 1996 (Schappert, 1998). The 
table shows the distribution of all visits made to ambulatory care settings by patients with 
different insurance arrangements in 1996. 

Of all ambulatory care visits paid by FFS insurance, 10.3 percent were to EDs, 6.1 percent 
to outpatient departments, and 83.6 percent to physicians’ offices. In contrast, patients with 
expected payment under a commercial HMO/prepaid system were less likely to use the ED— 
only 6.8 percent of all ambulatory visits for this group were to the ED and 87.1 percent were 
made to physicians’ offices. Ambulatory care visits by patients covered under commercial 
PPOs, a less stringent form of managed care, had only 6.6 percent of visits made to the ED and 
nearly 90 percent to physicians’ offices. Patients under commercial managed care 
arrangements had relatively lower rates of ED utilization compared to their FFS counterparts. 
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Table 2-4. Use of Ambulatory Care by Insurance Status, 1996 


Emergency 

Outpatient 

Physician 


Departments 

Departments 

Offices 

Total 

N=90,347 

N=66,186 

N=734,493 

N=892,025 

A, . 

% n 

% n 

% 


Private 


FFS 

10.3% 

(.56) 

18,521 

6.1% 

10,951 

83.6% 

150,571 

100% 

PIMO/Prepaid 

6.8% 

(.58) 

7,770 

6.0% 

6,853 

87.1% 

99,891 

99.9% 

PPO 

6.6% 

(.70) 

5,059 

3.1% 

2,419 

89.8% 

69,042 

99.5% 


Medicaid 


FFS 

imuM 

12,739 

20.4% 

12,765 

58.6% 

36,725 

99.3% 

PIMO/Prepaid 

13.4% 

(1.64) 

3,524 

13.8% 

3,628 

72.8% 

19,097 

100% 


SOURCE: Schappert (1998). N and n are in thousands; row totals may not sum to 100% due to 
rounding. SE calculated using combined settings coefficients. 


Medicaid patients had similar tendencies. Patients with FFS Medicaid had higher rates of 
ED utilization (20.3% of ambulatory care visits for the group) compared to those under HMO 
Medicaid arrangements (13.4%). Findings from both private and Medicaid sources show that 
ED utilization relative to the rest of ambulatory care visits is highest under FFS as compared to 
managed care. Such findings are consistent with managed care’s objective of reducing ED 
visits and increasing primary care visits. 

Though these data provide some evidence that ED utilization is lower under managed care, 
there are three important caveats. First, the reliability of NHAMCS and NAMCS data on 
expected source of payment is unknown. Coders were asked to specify the kind of insurance 
(private vs. Medicare, Medicaid, etc.) and the type of payment (PPO, PIMO, FFS, self-pay, no 
charge, and other) associated with the medical visit. Much of the data, however, are missing for 
type of payment - 13 percent of privately insured individuals and 36 percent of Medicaid 
patients had no type of payment specified. NPIAMCS/NAMCS attempted to capture better data 
in 1997 with redesigned variables, but this also resulted in substantial missing data. Central to 
this issue seems to be difficulty in achieving consensus on the coding of diverse managed care 
arrangements. For example, though no formal PPOs are in place under Medicare and 
Medicaid, some states may utilize a similar system for managing costs, and some data 
collectors have coded this. In spite of its limitations, NPIAMCS/NAMCS data were included in 
this report because there is no other source that provides the same richness of information 
about medical visits across emergency and ambulatory sites at a national level. Furthermore, 
the data illustrate the difficulties involved in trying to determine how managed care affects ED 
utilization. 
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While the findings in Table 2-4 are consistent with the successful implementation of 
managed care strategies to decrease ED utilization, there are several explanations that could 
account for the relatively low rate of ED visits under managed care. HMOs expend 
considerable effort and seem to be successful in increasing primary care contact with their 
members (Tu, Kemper, & Wong, 1999). The differences in the patterns of ambulatory care 
utilization may be due to increased primary care contact, rather than decreased ED utilization. 
Managed care organizations have traditionally attracted and actively recruited individuals who 
are healthier, and differences in the proportion of ED visits may be due to selection bias (or 
adverse selection) that leads to higher utilization for FFS patients. Unfortunately, NFIAMCS and 
NAMCS provide little information on client histories and general health, which would permit 
analyses sensitive to case mix. 

A related issue is whether the utilization pattern described above is attributable to managed 
care’s systematic diversion of non-urgent cases from the ED. If the lower ED utilization rate is 
due to the implementation of managed care practices diverting nonurgent cases from the ED, 
then most of the ED cases under managed care will be urgent/emergent cases. Similarly, 
managed care diversion practices should result in a higher proportion of urgent/emergent cases 
when compared to FFS cases. Table 2-5 shows the proportion of NFIAMCS ED 
urgent/emergent visits, by type of insurance, for 1993 and 1996. The trend data in this table 
must be interpreted with caution, since the NFIAMCS insurance categories changed markedly 
between 1993 and 1996. Nevertheless, it is likely that the relationships between FIMO-covered 
visits and FFS-covered visits within each year are relatively robust. 

Contrary to expectation, there was no difference in 1993 between commercial managed 
care and FFS cases in terms of the urgency of the condition for which they were treated. This 
could be because managed care practices had no effect on the acuity of conditions treated in 
the ED. Alternatively, the statistical power for detecting the effect may be too low, given the 
amount of variability contributed by other factors and the low number of cases for which 
managed care was the expected source of payment. 

Flowever, in 1996, there was a statistically significant difference in the acuity of cases 
treated under private managed care compared to acuity in FFS cases. Specifically, cases 
covered by private FiMO arrangements were more likely to be coded as “urgent” than were 
those under private FFS arrangements. Thus, these data suggest a relationship between 
managed care and ED utilization, with less urgent cases being diverted from the ED to other 
settings. 
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Table 2-5. Urgent/ 

Emergent ED Visits by Type of Insurance, 1993 and 1996 

TYPE OF 
INSURANCE 

Percent of ED Visits Labeled 
“Urgent,” 1993 

Percent of ED Visits Labeled 
“Urgent,” 1996 


Percent 

(SE) 

N 

Percent 

(SE) 

N 

Private HMO 

45.2% 

(1.4) 

6,421,194 

47.2% 

(1.01) 

15,450,000 

Private FFS 

43.9% 

(.65) 

28,440,000 

44.5% 

(.92) 

18,430,000 

Medicaid HMO 

46.5% 

(10.6) 

50,453 

40.1% 

(2.07) 

3,542,202 

Medicaid FFS 

38.8% 

(2.4) 

8,100,442 

44.0% 

(1.0) 

15,730,000 

Self Pay 

41.5% 

(1.0) 

11,690,000 

42.8% 

(1.0) 

15,190,000 


SOURCE: analysis of NHAMCS public-use data files 


The interpretation of findings for Medicaid data is less straightforward. There were no 
significant differences in acuity of patients covered by either HMO or FFS payment systems. In 
1993, there were too few patients in the Medicaid FIMO category to create enough statistical 
power. In 1996, there was a tendency for Medicaid FIMO cases to be less urgent than FFS 
cases. Though this difference did not quite reach statistical significance, it may be indicative of 
a selection effect - states have tended to enroll their healthiest Medicaid patients in FIMOs as 
opposed to FFS programs (Zuckerman, Evans & Flolahan, 1997). 

In these analyses, self-pay cases had the lowest level of urgent/emergent status. This is 
consistent with assertions that persons without insurance use the ED for primary care and less 
urgent conditions. 


Findings from MEPS 

To explore further the relationship between managed care and ED utilization, data from the 
1987 and 1996 MEPS were analyzed. Whereas NHAMCS provides information about the 
population utilizing the ED and other ambulatory services, MEPS provides information about 
insurance coverage and health care utilization patterns for the general population. 

As shown in Table 2-6, patients with HMO-type insurance were significantly less likely to 
use the ED in 1987 than individuals with non-HMO insurance (pc.001). Although the rates of 
ED utilization decreased for both HMO and non-HMO populations over the next 9 years, the 
between-group difference remained significant in 1996 (p<.05). The differences in ED utilization 
rates, however, may be due in some part to a selection bias (HMO patients may have a lower 
ED utilization rate due to better overall health status). No attempt was made to account for 
differences in health status between the two groups. 
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Table 2-6. ED Utilization by HMO Status for Insured Persons, 1987 and 1996 (MEPS) 



1987 

1996 


HMO 

Non-HMO 

HMO 

Non-HMO 

Mean Number ED Visits 

.20 

.26 

.17 

.19 

% Persons With One or More ED 
visits 

14.6% 

17.5% 

12.6% 

13.4% 

Mean Number ED Visits Among 
Persons With One or More ED 

Visits 

1.34 

1.47 

1.35 

1.40 


SOURCE: analysis of MEPS public-use data files 


In summary, findings from both MEPS and NHAMCS demonstrate that lack of insurance 
and managed care affect the ED population from which DAWN detects its cases. However, 
interpretations of these findings are equivocal because of changes in NHAMCS and MEPS 
variables over time, possible selection bias effects in managed care, and diverse and rapidly 
changing managed care practices. Furthermore, because the size of the effect is unknown, this 
research yields no conclusive evidence that one or more health system changes have 
systematically and consistently affected EDs nationwide in ways that are consequential for 
DAWN. 

The inconclusive literature review did not dispel lingering concern about the amount, 
source, and impact of “noise” being introduced into the DAWN data by factors related to health 
system change. In particular, little is known about how the highly variable number of alternative 
medical sites and the characteristics of the managed care market affect persons with substance 
abuse problems who might seek help in the local ED. It was unknown whether data collection 
for DAWN from those alternative sites was either warranted or feasible. For this reason, a 
qualitative study of the health care system in four cities was conducted. ED staff, emergency 
services staff, and other health care providers contributed information about where persons with 
substance abuse problems seek health care. This study is described in detail in the following 
section. 


QUALITATIVE RESEARCH ON LOCAL HEALTH SYSTEM IMPACTS 

A qualitative study was conducted to describe the role of the ED in the treatment of 
substance abuse problems and the changing factors that influence whether patients present for 
care there or elsewhere. To accomplish this, a series of focus groups was convened to assess 
what recent changes have occurred and are anticipated in the organization and location of 
emergency care for substance abuse. The focus groups included representatives from 
throughout the health care system. Discussion concentrated on the impact of health system 
changes on the use of hospital EDs to treat acute substance abuse problems. The resulting 
qualitative data about the nature and variation of health system change across several cities are 
to be used to guide decisions about whether to expand DAWN data collection beyond EDs to 
improve the validity of DAWN as an indicator of drug-related morbidity. Information gathered in 
focus groups and interviews conducted in four metropolitan areas is used to describe where the 
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kinds of drug-related problems captured by DAWN are treated. Factors that determine where a 
person receives medical care are identified, along with the potential for these factors to change 
over time. 

The potential for increasing “leakage” or “flooding” of cases is discussed. Leakage of 
substance abuse cases from the ED (and from DAWN) occurs when cases usually treated in 
the ED are diverted to other health care settings as a result of changes in the health care 
system (recent growth of managed care and urgent care centers). Flooding of cases into the 
ED occurs when these changes shift cases usually treated in alternate settings into the ED. 


Procedure 

Key informant interviews and five focus groups were conducted. Unstructured key 
informant interviews were conducted initially to identify focus group participants and to 
complement the information gathered in focus groups. Key informants were asked about the 
characteristics of persons with substance abuse problems seen in their facilities, what barriers 
exist to care for this population, and what changes in access have occurred. Names of key 
informants and focus group members were generated using the snowball sampling technique; 
each informant was asked what other individuals could provide important viewpoints regarding 
substance abuse problems and treatment in their community. Special efforts were made to talk 
to individuals whose names were mentioned frequently by previous informants. 

Because a comprehensive view of access to health care by persons with substance 
abuse problems was needed, a broad range of disciplines and professions was included in this 
study. Focus group members and key informants were drawn from the following professions: 
police officers (particularly those involved in community policing); emergency medical services 
personnel (fire and rescue, paramedics, emergency medical technicians); substance abuse 
treatment providers; employee assistance professionals (EAPs); ED staff (including nurses, 
doctors, and managers); medical staff from community clinics; social workers in medical clinics 
and EDs; urgent care/triage staff; substance abuse prevention/harm reduction workers; 
psychiatric ED/crisis center staff; and public health administrators. 

The first focus group involved ED physicians at an annual conference of the American 
College of Emergency Physicians (ACEP). Since many of the participants were administrators 
as well as clinicians, and because they were involved and interested in public health issues at a 
national level, this group provided an overview of issues concerning substance abuse in EDs 
across the country. The questions asked of these experts included the following: 

■ Flow does substance abuse or the use of illicit substances present in the ED? Are there 
problems related to substance abuse that are of particular importance due to severity 
of the illness? Are there particular kinds of cases that you feel should be monitored for 
public health reasons? Why? 

■ What factors affect whether persons with substance abuse problems are seen in the 
ED, as opposed to other health care sites? Plave these factors changed? Flow has 
managed care affected the population you see in your ED? Plas managed care 
affected the number or characteristics of persons with substance abuse problems you 
see? Why or why not? 
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An additional focus group was conducted with a range of professionals in each of four cities 
(Seattle, Phoenix, Miami, and Boston). These cities were chosen because they were 
geographically dispersed and because background information on their respective health care 
systems was available through ongoing work by the Center for Studying Health System 
Change. Persons from different organizations in the community were contacted because 
substance abuse presents differently, depending on where in the community a person is 
seeking help. For example, police and emergency medical services personnel have frequent 
contact with persons who are intoxicated or unconscious from substance use; medical social 
workers and clinic staff are more aware of the chronic medical problems associated with abuse. 

In order to ensure that focus group respondents were discussing the same subject, they 
were asked questions about how persons with particular substance abuse-related problems 
access care. These problems constitute most of the substance abuse-related problems seen in 
EDs and detected in DAWN. They include drug overdoses, seizures, chest pain, injuries related 
to fights or accidents, psychiatric emergencies (e.g., hallucinations, depression), intoxication, 
conditions related to injection drug use (e.g., abscesses, cellulitis), organ damage related to 
substance abuse (e.g., liver damage, pancreatitis), withdrawal, and requests for substance 
abuse detoxification or treatment. 

Respondents were asked to identify the settings that treat each of these substance abuse 
problems in their community. Problems involved alcohol as well as drugs, and sites included 
those providing mental health and substance abuse treatment as well as medical treatment. 
Respondents were asked the following questions: 

■ How and why do people seek help for this substance abuse problem? 

■ What setting is contacted? Where do the patients/clients go? What factors determine 
where a person goes for care? 

■ Where does the ED fit into the range of health care options available for persons with a 
substance abuse problem? What factors determine whether a person goes to the ED 
or to an alternative setting? 

■ Who in your community would have an important perspective on issues related to 
access to health care for people with substance abuse problems? 


Results: Emergency Physicians Focus Group 

ED physicians at the annual ACEP conference made several important points. First, they 
noted that the most widespread source of health system change, managed care, has not 
decreased ED caseloads recently. In fact, managed care may have increased visits to the ED 
because primary care visits are closely scheduled under managed care, leaving limited space 
for patients who required same day or even next day treatment. 
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Second, respondents did not expect that managed care practices would affect ED visits 
related to substance abuse for several reasons: 

(a) There is limited access to primary care or urgent care providers late in the evening or 
on weekends. The ED is still the site of choice during those hours when most problems 
related to drug use occur. 

(b) Because managed care dictates what provider a patient may see, many patients are 
forced to switch providers. These forced changes lead to less reliance on their primary 
provider and more incentive to seek care at the ED. 

(c) The ED may be preferred by patients who do not want their primary care provider to 
know of their substance abuse problem. 


Results: Metro Area Focus Groups 

Qualitative study of four metropolitan areas revealed that the ED plays an important role in 
the treatment of substance abuse problems in every community. Different kinds of substance 
abuse problems are routed through the health care system in very different ways, leading to 
differences in vulnerability to diversion to or from the ED. Substance abuse conditions treated 
in the ED can be grouped in four categories: 

(1) Urgent medical conditions 

(2) Psychiatric emergencies 

(3) Less urgent medical conditions 

(4) Visits related to public health 


Urgent Medical Conditions 

There was clear consensus that the vast majority of urgent medical problems were sent to 
and treated almost exclusively in the ED, and that this practice was stable across cities and over 
time. The most urgent substance abuse-related problems were identified as overdoses, chest 
pain, seizures, and major injuries/accidents (including threat to life or long-term normal 
functioning). Urgent conditions such as these are rarely diverted from the ED. Therefore, the 
number of urgent medical cases is likely to provide a valid indicator of the level of drug-related 
morbidity for this group of conditions. 


Psychiatric Emergencies 

In addition to treating urgent medical conditions, the ED plays an important role in the 
treatment of psychiatric emergencies. In the cities studied, most of the public health systems 
rely on the ED for medical clearance before an on-site psychiatric assessment. However, some 
managed care companies and one city provided clearance and assessment in independent 
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psychiatric facilities. Also, it was found that the role of the ED in treating psychiatric 
emergencies has been variable over time. In the private sector, access to alternative 
psychiatric sites changes according to the contractual arrangements of managed care with 
employers and with the sites themselves. In the public sector, access to freestanding 
emergency psychiatric facilities changes according to public health funding. Substance abuse 
cases involving psychiatric emergencies seem to move into and out of the ED as managed care 
and public health policies change. 


Less Urgent Conditions 

The ED and alternative sites also treats a number of other substance abuse-related 
problems that are universally regarded to be less urgent. The problems identified as less urgent 
are minor injuries/accidents, organ damage/chronic illnesses related to drug use (e.g., 
pancreatitis, hepatitis, etc.), illnesses related to the administration of drugs (e.g., cellulitis, 
abscesses, etc.), withdrawal, and request for substance abuse treatment or detoxification. 

Because the ED is not the primary setting for treating less urgent drug-related conditions, 
the number of ED cases depends on the perceived acuity of the condition and access to other 
health care sites. When treatment at other health care settings is unavailable or inconvenient, 
people come to the ED. Access to alternative sites varies across cities and over time due to 
local factors that are numerous and rapidly changing. Though data collection was limited to four 
cities, the study revealed six factors affecting ED utilization for less urgent conditions associated 
with substance abuse: 

Availability of publicly funded primary care sites for the uninsured . Since most primary care 
sites will not treat people without insurance, many uninsured people seek care in EDs, which 
are required by federal law to provide evaluations and basic treatment. Some cities set up 
primary care clinics to treat the uninsured, but the number of these clinics and the level of 
funding vary between cities and over time. For example, Boston and Seattle appear to have 
well-developed systems serving this population. Miami has some specific clinics available for 
the uninsured and indigent, but Phoenix seems to have a poorly developed infrastructure to 
serve this population. A relatively large number of substance abuse cases without insurance 
are treated in community clinics in Boston and Seattle. In contrast, because Phoenix reportedly 
lacks an adequate system for serving the large number of medically indigent persons in that 
city, the ED is flooded with nonurgent cases, including those involving substance abuse. 

Police procedures . Local law enforcement policies determine whether and how intoxicated 
persons encounter the health care system in the course of police processing for crimes. For 
example, Miami police require evaluation of persons with an altered mental status at the ED 
before transport to jail, but Phoenix does not. EDs in cities where police are required to obtain 
medical clearance for intoxicated persons are likely to be “flooded” with intoxication cases. The 
vast majority of these cases is likely to be intoxicated with alcohol. As police procedures 
change, the number of intoxicated persons seen in the ED will change. 

Managed care policies . Reports from the Center for Studying Health System Change and 
conversations with key informants suggest that the face of managed care varies across metro 
areas and over time. This variation is related to complex factors such as market competition, 
state legislation, employer preferences, and the status of providers. 
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Rather than decreasing ED utilization, managed care is now blamed in several cities for 
poor access to primary care doctors. Physicians are booked too tightly to allow insured people 
to be seen without a long wait. The delay for treatment has increased in all four cities over the 
last 10 years, but delays still vary across cities (e.g., delays in Phoenix are longer than in 
Boston). Increasingly, people with managed care insurance use the ED because they are 
unable to see a primary care provider in a timely manner. Rather than diverting cases from the 
ED (see the analysis of 1996 NHAMCS data above) in many cities managed care organizations 
seem to be responsible for the increased use of EDs by insured persons for less urgent 
conditions. 

Access to urgent care. Urgent care exists in three different forms: as extended hours in 
primary care offices, as an adjunct unit to the ED, and within stand-alone clinics. Different forms 
of urgent care have waxed and waned in the four cities over time - the effects on EDs have 
been variable over time across cities. For example, a Boston HMO has gradually restricted 
hours for urgent care in its primary care sites, and the number of stand-alone urgent care 
centers has plummeted in the last 10 years. A Seattle HMO seems to be increasing access to 
urgent care in a well-to-do suburb. Urgent care sites unaffiliated with EDs treat mainly the 
privately insured, and they have a reputation for turning away all others. Urgent care staff 
reported that substance abuse is rarely identified or documented in their facilities, for several 
reasons. First, the less critical injuries seen in urgent care are usually treated many hours after 
the injury event, when any trace of a drug is likely to be gone from the patient’s system. 

Second, the focus of urgent care clinicians is the rapid treatment of acute conditions. Questions 
about substance use are not typically asked, and if substance use is mentioned, it is often not 
documented because it is not perceived to be relevant for the treatment of the acute condition. 
Third, for those urgent care sites unaffiliated with the primary care clinic, medical records are 
not typically available, and information about historical symptoms, which raise-questions of 
undiagnosed substance abuse, cannot be examined. 

Urgent care may be responsible for leakage of some nonurgent problems from the ED, but 
the extent of the leakage varies across cities and its total volume is unknown. In addition, two 
practical issues make DAWN data collection from these sites infeasible for future study: rapid 
changes in the location, structure, and operation of specific facilities; and poor identification and 
documentation of substance abuse in the records of persons treated. 

Availability of needle exchange programs. Some cities have established needle exchange 
and other harm reduction programs that treat nonurgent drug abuse and other health problems 
that would otherwise be seen in the ED. For some IV drug users, these programs are their only 
contact with any segment of the health care system. Focus groups included representatives 
from needle exchange programs in Boston and Seattle; no such program is available in 
Phoenix, and none was mentioned by the Miami respondents. Where these programs exist, 
they are likely to reduce the number of IV drug abusers accessing emergency care. However, 
needle exchange programs are highly susceptible to local political agendas and will vary across 
geographic areas and over time. 

Access to substance abuse treatment. Informants in all four cities noted that there have 
been fewer resources available for substance abuse treatment recently, as a result of 
decreases in benefit coverage by private insurance and a decrease in detox facilities. However, 
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the degree of burden experienced by the EDs as a result of these changes seems to vary 
considerably. Individuals in Phoenix report a crisis, where substance abusers come in and out 
of the ED repeatedly with no improvement in their addiction problem. In that city, respondents 
clearly linked lack of services to higher ED visits related to substance abuse. In contrast, 

Boston EDs seem to be coping more successfully with the increased burden. 

In summary, the ED and many other sites provide care for less urgent conditions 
associated with substance abuse. Substance abuse cases are susceptible to diversion 
(leakage) or flooding. The degree of diversion or flooding varies across cities and over time 
according to a number of factors. Moreover, it is not readily feasible to capture nonurgent 
substance abuse cases for surveillance through the expansion of DAWN data collection beyond 
EDs for three reasons: (1) the large number of alternative sites; (2) the high variation in the 
number and characteristics of alternative sites between cities; and (3) the constant change in 
resource allocation that affects patients’ access to these sites. 


ED Visits Related to Public Health Policy 

In addition to the treatment of medical conditions, the ED serves as a point of access for 
public health services for two groups of patients - homeless, inebriated persons, and persons 
seeking detox treatment. The degree to which the ED is central to public health management of 
these patients varies across cities and over time. 

Administrative policies for public sector detox treatment. The role of the ED in the delivery 
of public sector detox services seems to vary significantly among cities. In Boston, patients are 
required to obtain medical clearance from the ED prior to entering a public detox program. 
Patients in Miami and Phoenix are evaluated at the detox center first; and only those patients 
meeting certain criteria are sent to the ED. This difference in administrative procedures is likely 
to result in a higher number of low morbidity, substance abuse cases in certain cities. Changes 
in administrative procedure would quickly lead to the diversion of these cases to other sites 
designated by the public health system. 

Public health policy for homeless, inebriated persons . People who are homeless and who 
are chronically intoxicated often seek care in the ED for a variety of social problems rather than 
medical conditions associated with substance abuse. In Boston, one of the roles of the ED is to 
provide access to social services, and intoxicated persons who are “found down” by emergency 
services personnel are taken to the ED. In Phoenix, a special van is called to take intoxicated 
persons to a detox center. However, this van service is available only where there are beds 
available, and the public detox facility is often full. In Seattle, an intoxicated person can be 
taken directly to a shelter or to a “sobering center” especially designed to divert these cases 
from the ED. The likelihood of diversion varies over time, and the specific places to which 
patients are diverted differ across cities and over time. The availability of sites other than the ED 
that provide treatment for public intoxication and its related social problems appears related to 
resource allocation at the county level. 

Informants indicated that the number of cases involving intoxicated individuals seen in the 
ED is high, and the vast majority of these cases are intoxicated with alcohol alone. This issue is 
problematic for the surveillance of alcohol abuse in EDs. Because the number of these cases 
seen in the ED is determined by local policies, the number of alcohol-related cases reported for 
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any given community will be a function of the manner in which the community manages both 
homelessness and public intoxication. 


DISCUSSION 

This qualitative study suggests that health care system change is ongoing and rapidly 
shifting. Managed care, which in the mid-1990s was thought to be responsible for decreased 
ED utilization related to diversion practices, now seems to be responsible for increased 
utilization related to inadequate access to primary care. Health system change also seems to 
be highly localized, as evidenced by the large number of local factors that can determine ED 
access. Thus, there is no single source of change that has affected all DAWN sites in the same 
way, and it is not feasible to collect data from any single site or group of sites that would 
quantify the “leakage” of cases from DAWN. 

However, because cases associated with different substance abuse conditions are 
differentially susceptible to leakage and local effects, it makes sense to include data elements in 
DAWN to differentiate among these conditions. The redesign team has recommended changes 
to the variables collected about each DAWN case. At a minimum, data elements should include 
either diagnosis or presenting complaint, along with the disposition of the case. Presenting 
complaint might be further refined to focus on the major injury/illness categories noted earlier in 
this report. Together, these data elements can assist analysts in understanding the specific 
conditions/reasons for which substance abusers seek care in EDs, as well as the relative 
urgency or severity of the different conditions reported. 

This study was designed to gather qualitative information about the factors affecting ED 
access in a few select cities. The approach cannot quantify the size of the population of cases 
subject to leakage from EDs. Because of the large number of alcohol intoxication cases seen in 
many EDs, the less urgent cases are likely to outnumber the more urgent cases. Excluding 
alcohol intoxication cases, it is unclear whether the remaining, less urgent conditions outnumber 
the urgent conditions. However, the data suggest that a substantial proportion of the less 
urgent, substance abuse-related cases are seen in sites other than the ED, and that there are 
clearly identifiable local factors that determine the degree of diversion (or leakage) to alternate 
treatment sites. The effects of these factors vary over time and location. 

It seems clear that the ED plays an important role in the treatment of substance abuse 
problems in every community studied. While the ED is the sole setting for the treatment of 
urgent conditions, there are other settings that treat less urgent conditions associated with 
substance abuse. As access to these alternative sites changes according to local factors 
(health care policy, market changes, and funding), cases associated with less urgent conditions 
are subject to leakage or flooding. While monitoring the number of substance abuse cases 
seen in the ED can provide important information about the physical consequences of drug 
abuse, the expansion of DAWN data collection to other health care settings is not 
recommended. DAWN would benefit, however, from collection of information that would allow 
analysts to distinguish conditions that are differentially susceptible to leakage and the effects of 
local health system change. DAWN cannot measure the prevalence of all drug-related 
illnesses, but available evidence suggests that DAWN is a valid indicator of the nation’s 
substance abuse problem. 
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3. DEVELOPMENT OF A NEW SAMPLE DESIGN 
FOR DAWN 


A core component of the development of an alternative design for the Drug Abuse Warning 
Network (DAWN) included the development of specifications and methods for selecting a 
new sample for the emergency department (ED) component. 1 As noted in Chapter 1, the 
1997 Office of Applied Studies (OAS) review panel and subsequent constituent analyses by the 
redesign team raised questions about the appropriateness of EDs for data collection, the 
efficiency of the current sample design, and the quality of estimates produced (notably, the large 
weights assigned to a number of relatively small facilities, and the relative standard errors 
obtained). In addition, questions were raised about the possibility of expanding DAWN to 
encompass new metropolitan areas or rural areas. 

Chapter 2 addressed the impact of health system change on the emergency care system 
and its relevance for DAWN. Based on the conclusions of that extensive research, the redesign 
team and OAS jointly concluded that continued focus on EDs was warranted. Changes were 
recommended in other aspects of the redesign (e.g., case definition and data elements) that 
would enhance the information obtained from those facilities and ensure a better fit between 
information gathered and the constraints imposed by the settings in which DAWN operates. 

From a sample design perspective, however, it was determined that DAWN should continue to 
focus exclusively on general medical, non-Federal, short-stay hospitals operating 24-hour EDs. 
Within that general constraint, however, there were numerous possibilities for adapting the 
sample design to meet demands from the users of DAWN for geographic coverage and 
statistical precision. 

This chapter reviews the considerations, constraints, and recommendations resulting from 
activities focused on redesigning the ED sample for DAWN. At the outset, it should be noted 
that the redesign team was tasked with developing recommendations for the general 
parameters of the new sample and a method for drawing the actual sample. The specific 
facilities that will constitute the new sample, however, were not identified by the redesign team. 
Because the population of EDs changes rapidly from year to year (due to acquisitions, changes 
in ownership, mergers, demergers, and closures), OAS requested a method for sample 
selection, with the actual selection to be undertaken by the DAWN analytic contractor during the 
implementation of the new DAWN design. This chapter presents an overview of 
recommendations and a strategy for drawing a new sample for DAWN. All sample size 
requirements are approximate pending actual selection of the final sample. 


1 Recall that the medical examiner component is not based on a sample of facilities, and the redesign team did not recommend that 
the DAWN-ME component be restructured to generate statistical estimates. Therefore, this chapter deals only with sampling and 
estimation strategies for the ED component of DAWN. 
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OVERVIEW 


DAWN monitors changes in drug abuse trends in the United States and within particular 
metropolitan statistical areas (MSAs). Data are gathered from a representative sample of EDs, 
as well as from a group of participating medical examiners and coroners (ME/Cs). Estimates of 
drug-related morbidity (illness) are produced by collecting data on drug-related hospital ED 
visits, including the particular drug(s) involved, and assuming that changes in the drug-related 
ED visit estimates are indicators of similar changes in drug abuse trends in general. 

The hospital EDs included in the DAWN data collection are selected through a probability 
sample. DAWN currently uses a stratified, single stage sample of general medical, non-Federal, 
short-stay hospitals with 24-hour EDs and collects data on all drug-related ED visits (i.e., 
episodes) within the sampled hospitals. The current sample was drawn in 1988 and with respect 
to particular analytic objectives and precision requirements. Although new hospitals are allowed 
a chance of selection through an annual sample maintenance process, most of the sampled 
units and efficiency of the sample design date back to the 1988 sample selection. 

DAWN currently produces annual and semi-annual estimates of the number of drug-related 
ED visits (episodes) and drug mentions for the coterminous United States (i.e., excluding Alaska 
and Hawaii) as well as for 21 specific MSAs. These estimates are obtained through a probability 
sample of hospitals selected to be representative of each of the 21 MSAs, yielding about 470 
responding hospitals as of 2000. The data obtained from these facilities are weighted to 
produce estimates of the total number of drug-related ED visits and drug mentions in each MSA. 
Some MSAs have lower relative standard errors (RSEs) for their estimates than others. 

Among the participating hospitals are about 110 facilities that were selected to represent 
that portion of the coterminous United States that lies outside the 21 MSAs. These are referred 
to as “national panel” hospitals. In 2000, 83 of the 110 sampled national panel hospitals 
provided data for DAWN. The data obtained from these hospitals are weighted to produce 
estimates of the total number of drug mentions and drug-related visits in the remainder of the 
United States. These 110 hospitals are weighted to represent the 4,000 or so hospitals in the 
areas outside the 21 oversampled MSAs. For this reason, the average weight assigned to each 
of the national panel hospitals is about 50 (and sometimes higher). This means, for example, 
that a national panel hospital with 2 methamphetamine mentions in 2000 could contribute 100 
(2x50) mentions to the national estimate for the year 2000. This issue caused considerable 
concern for analysts familiar with the DAWN data. The weighting issue is particularly 
problematic because DAWN seeks to measure rare events that may not be randomly distributed 
across the universe of hospitals. Adding more hospitals to the national panel would reduce their 
weights and improve the estimates they produce, but this would also increase the cost of data 
collection. 

The overall estimate for the coterminous United States is derived by adding each of the 
MSA estimates to the national panel estimate. On balance, this approach yields an estimate 
with an acceptable level of statistical error (i.e., meeting or exceeding the precision target levels 
established for DAWN’S sample redesign in 1988—see “DAWN Sample Design and Estimation 
Procedures” [OAS, 1997]). 

The redesign team’s constituent analyses suggested some support, particularly among the 
Federal agencies represented, for DAWN’S continued production of national estimates. The 
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primary users of these data would be Federal agencies interested in tracking overall, year-to- 
year trends in drug mentions or drug-related visits nationwide. In addition, SAMHSA’s legislative 
mandate—although not explicit—implies that the agency will produce a count or estimate of the 
number of drug-related ED visits nationwide each year. 

In March 2000, the redesign team’s Technical Advisory Group (TAG) and the OAS 
Director’s advisory group considered whether DAWN should continue to produce national 
estimates, as well as the optimal level of analysis for subnational estimation (e.g., MSA, state, 
county). Members of the advisory groups indicated an interest in national estimates but noted 
that they lacked confidence in the current DAWN estimates, which resulted in under-utilization 
of these data. Agencies expressing a particular interest in the national data were the Office of 
National Drug Control Policy (ONDCP) and the Drug Enforcement Administration (DEA). The 
major reasons expressed for lack of confidence in the data were the large weights assigned to 
many of the national panel facilities, and the substantial relative standard errors associated with 
estimates of emerging drugs such as methamphetamine. Overall, panelists encouraged 
continued work to improve the efficiency of the ED sample and estimation procedures. 
Subsequent discussions urged the redesign team to consider expanding the DAWN sample to 
provide truly national coverage (i.e., including Alaska and Hawaii), as distinct from the coverage 
of the coterminous United States. 

Additional findings from the redesign’s constituent analysis indicated that the majority of 
DAWN’S users and applications are at the local level (city/county/MSA/state). Therefore, it was 
agreed that an alternative design must consider how best to generate locally relevant data. 
These considerations included—but were not limited to—the number, type, and distribution of 
areas for which DAWN provides data. Thus, geographic distribution of the sample also factored 
into the redesign process. 

Given the age of the sample, the related efficiency issues and changes in the general 
population, OAS called for an evaluation of the current DAWN sample design. This request was 
motivated by the following specific concerns: 

■ Changes in the hospital and ED population over time. 

■ Changes in the drug abusing population and related analytic objectives. 

■ The current performance of the sample relative to MSA and drug specific estimates. 

Hospital characteristics have changed considerably since the current sample was 
implemented in 1988, and especially with regard to characteristics that were used in the original 
sample design and stratification. The characteristics used in the original sample design 
included a hospital’s annual count of ED visits and the presence or absence of an 
alcohol/chemical dependency unit or an outpatient department at the hospital in which the ED 
was located. The drug abusing population has also changed considerably with the emergence 
of new drugs, the geographic spread of existing drugs, and the demographics of drug users. 

The statement of work for the DAWN sample redesign acknowledged that addressing these 
concerns could result in many changes to the current sample design. In particular, the 
statement of work called for new research into the following five general areas: 
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■ Analytic objectives that could and should be achieved; 

■ Total sample size required for those objectives; 

■ National panel sample size required; 

■ List of MSAs specifically targeted; and 

■ Stratification of sampling units to be used. 

The total sample size required and the national panel sample size required are a function of 
the analytic objectives. These objectives cover the estimates of interest, the precision required 
of these estimates, and the levels (nationally, by MSA) at which estimates are required. While 
these objectives tend to increase sample size as they grow more ambitious, stratification and 
optimal allocation can work to decrease the required sample sizes. The redesign team 
developed a sample design optimization process to satisfy these objectives simultaneously. 

This report presents the results of that process. 

The redesign team carried out the sample design evaluation and research in two separate 
steps. First, the original sample design and its analytic objectives and precision requirements 
were documented. This was necessary because the original design, its required sample size, 
and its performance would serve as the standard for comparison with all alternative, new 
designs. Second, a sample design optimization process was developed that would minimize 
the sample size required to achieve a set of analytic objectives, given a particular sample 
design. This was accomplished by expressing the problem as a mathematical programming 
problem. 

The combination of analytic objectives and sample design were referred to as “sample 
design scenarios.” Various analytic objectives that were considered included estimation at the 
national and MSA level for as many as six different estimates (ED visits, total drug-related 
episodes, and episodes involving marijuana, cocaine, heroin, and/or methamphetamine) all with 
minimum RSE requirements. Sample designs included stratification by one or more of the 
following: MSA, ownership (public, non-public), and size (based on annual ED visits). 

Using various combinations of analytic objectives, stratification criteria, and precision 
requirements, the redesign team obtained minimum required sample sizes for over 100 sample 
design scenarios. These results allowed the Substance Abuse and Mental Health Services 
Administration (SAMHSA) to review and consider the resources required by a set of analytic 
objectives as well as to interpolate the analytic objectives that could be met by a different level 
of resources. An iterative process was used to define parameters, generate scenarios, obtain 
solutions, and review results. New scenarios changed the number of targeted MSAs, the 
precision level required, the combination of estimates desired at different precision levels, etc. 
This iterative process was successful in defining the basic requirements of the new DAWN 
sample design. More detailed presentation of the sample design optimization approach and the 
results are provided throughout this report. 

The proposed new design was obtained through iterations of specifying sample design 
scenarios and obtaining results from the sample design optimization process. At the end of this 
iterative process, OAS was able to identify a sample design that met its analytic goals within the 
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projected budget for the new implementation. The proposed new design has the following 
features: 

■ Requires an estimated 950 responding sample units; 

■ Includes 48 targeted MSAs; 

■ Provides national and MSA-specific estimates; 

■ Includes Alaska and Hawaii in the national estimate; 

■ Utilizes stratification by MSA, ownership, and size (ED visit volume); 

■ Provides RSEs <= 15% for national estimates; 

■ Provides RSEs <= 10% for MSA-specific estimates; and 

■ Includes estimates of total drug-related episodes, cocaine episodes, heroin episodes, 
and marijuana episodes at specified precision levels. 


Overview of This Document 

The remainder of this chapter provides more detail on the current DAWN sample design, 
the sample design optimization process, and the results from that process. The current sample 
design and its performance are described first. Next, the sample design parameters required 
for a redesign are reviewed. This is followed by a description of the models used to estimate 
counts of total episodes and drug-specific episodes at the hospital level for the entire hospital 
population. The sample design optimization process is then documented, along with the sample 
design scenarios and the results obtained from the optimization process. With this information 
as background, the proposed new design is then presented. The final section includes a 
number of sampling issues that were also considered and should be further evaluated as the 
new design is implemented. 


CURRENT SAMPLE DESIGN AND PERFORMANCE 

The original DAWN sample was drawn in the 1970s and consisted of approximately 800 
sampled units in 30 or more MSAs. The sample degraded over time and, by the mid-1980s, was 
considered out of date due to considerable nonresponse and nonrandom replacement of 
original selections. In 1988, a new probability sample was drawn for DAWN consisting of 730 
sampled units in 21 MSAs. This sample is still the basis for DAWN’S estimation and is updated 
annually to allow new hospitals a chance of selection. 

The current DAWN sample design consists of a stratified, single-stage cluster sample. The 
design uses 22 primary geographic strata with an additional 4 to 7 finer strata within each 
geographic stratum. The 22 geographic strata represent the 21 MSAs and the balance of the 
United States (the national panel). The stratification within each geographic area reflects the 
size of the hospitals (units with 80,000 or more annual ED visits were selected with certainty), 
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presence or absence of outpatient department or alcohol/chemical dependency units, and 
location within or outside the MSA’s central city. Hospitals are selected with equal probability 
within stratum and all ED visits, episodes, and mentions are captured within sampled hospitals. 
To date, 730 hospitals have been selected, of which 578 were still eligible and 466 were 
responding in 2000. Table 3-1 shows the total eligible population size, total sample size, eligible 
sample size, and responding sample size within each of the DAWN MSAs. 


Table 3-1. Population and Sample Sizes by MSA, 2000 


MSA 

Eligible 

Population 

Eligible 

Sample 

Responding 

Sample 

Atlanta 

32 

19 

15 

Baltimore 

21 

21 

21 

Boston 

45 

25 

22 

Buffalo 

10 

10 

8 

Chicago 

67 

37 

27 

Dallas 

31 

20 

11 

Denver 

14 

14 

9 

Detroit 

42 

21 

17 

Los Angeles 

86 

46 

34 

Miami-Hialeah 

24 

18 

16 

Minneapolis-St. Paul 

27 

18 

12 

New Orleans 

22 

16 

12 

New York 

78 

35 

31 

Newark 

25 

17 

16 

Philadelphia 

61 

32 

28 

Phoenix 

24 

18 

16 

St. Louis 

39 

29 

22 

San Diego 

20 

20 

18 

San Francisco 

19 

19 

16 

Seattle 

20 

15 

15 

Washington, D.C. 

31 

18 

17 

National panel 

3,950 

110 

83 

Total United States 

4,688 

578 

466 


Current Performance 

The current DAWN sample design was developed to satisfy certain precision requirements 
for national and MSA estimates (see Table 3-2). All precision requirements are expressed as 
RSE and with respect to estimates of total drug-related ED episodes. 


68 


Development of a New Design for DAWN 



Table 3-2. Precision Requirements for Original Sample Design 


Estimate RSE 

National 6% 

New York, Los Angeles, Chicago MSAs 6% 

All other MSAs (excluding 5 MSAs w/100% sampling) 8% 


Recent research indicates that some of these precision requirements are still being met by 
the current design; however, the performance varies by MSA and the precision for more specific 
estimates (such as cocaine episodes etc.) is lower for the national estimate and varies even 
more by MSA. Table 3-3 provides a summary of the performance of the current sample for 
certain estimates across MSAs in 1999. 


Table 3-3. Current Performance (RSEs) by MSA, 1999 


Estimate 

National 

Estimate 

Minimum 

(21 MSAs) 

Mean 

(21 MSAs) 

Maximum 

(21 MSAs) 

Total episodes 

7.3% 

0.4% 

9.8% 

21.3% 

Cocaine 

9.0% 

0.5% 

13.0% 

28.0% 

Heroin 

13.5% 

0.6% 

12.7% 

28.8% 

Marijuana 

11.8% 

0.4% 

16.6% 

41.7% 


Note that the minimum RSE values are associated with the five MSAs within which all 
hospitals are selected with certainty. 


SAMPLE DESIGN PARAMETERS REQUIRED FOR A REDESIGN 

In order to design a new sample, the redesign team required specific information on the 
following four general parameter classes: 

■ Level at which estimates are required (e.g., national, regional, state, MSA, urban vs. 
suburban, etc.); 

■ Specific estimates for which minimum precision is required (e.g., episodes, cocaine 
mentions, methamphetamine mentions, etc.); 

■ Precision measure (RSE, power, etc.) and performance level required; and 

■ Cost implications at each level of contact, recruiting, or collection (e.g., at the MSA, 
hospital, and case levels). 

These four parameters can occur in any combination. Any specific combination of 
parameters is referred to as a sample design scenario. Given decisions about the scenario 
preferred, a sample design can then be developed that either optimizes relative to these and 


Development of a New Design for DAWN 


69 







other constraints or that provides some balance or compromise between these competing 
constraints. 

Given projected resources and constituent demands, SAMHSA provided the following 
guidance for these parameters: 

■ Level - DAWN must provide national estimates as well as MSA estimates. DAWN will 
not provide regional or state-level estimates, nor will it provide estimates for areas 
smaller than MSAs. 

■ Estimates - DAWN must provide a national estimate of the total number of drug- 
related ED episodes, as well as reasonably precise estimates of the total number of 
drug mentions for cocaine, heroin, and marijuana. 2 

■ Precision - RSE levels of 20 percent, 15 percent, 10 percent, and 5 percent were 
initially selected for the national and MSA estimates, and for the four estimates listed 
above. 

■ Cost - The current cost and reimbursement model was provided by the current DAWN 
data collection contractor and was used to run a cost minimization model (described 
below). 


GEOGRAPHIC COVERAGE 

The number of targeted MSAs was identified early on as the principal driver in the 
estimated required minimum responding sample sizes obtained through the sample design 
optimization process. Given the importance of this particular design parameter, SAMHSA paid 
considerable attention to this detail, including a lengthy meeting in which SAMHSA and the 
redesign team reviewed maps, census data, and lists of targeted MSAs. The intention was to 
identify the particular list of MSAs to include in the final optimization runs and the proposed 
design. 

Population was a key factor in the decision about which MSAs should be included in the 
expanded DAWN sample. The 21 MSAs currently represented in DAWN are among the largest 
metropolitan areas in the nation, but they are not the 21 largest MSAs. For example, Houston is 
the largest MSA not currently represented in DAWN, while estimates are provided for Buffalo, 
which ranked 50th among MSAs in population in 1999. In order to best reflect the distribution of 
the U.S. population while maximizing the overall geographic dispersion of DAWN MSAs, two 
general options were considered: 

(1) Select MSAs in descending order of 1999 population, or 

(2) Select MSAs in descending order of 1999 population within census division. 


2 DAWN provides estimates for many hundreds of drugs but precision constraints considered in the redesign process centered on 
the major drugs of abuse. Inclusion of precision constraints for methamphetamine and other less common though 
epidemiologically interesting substances drove required sample sizes beyond the resources available. 
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Population figures for 1999 were the most recent available at the time the sample scenarios 
were generated. It was understood that, immediately prior to implementation of the 
recommended design, the DAWN analytic contractor would consult the latest available census 
data, and any changes to the list of MSAs would be reviewed with SAMHSA. 

In general, the objective was to maximize the percentage of the total U.S. population 
included in targeted areas while also obtaining considerable geographic dispersion of those 
areas. Several specific approaches were considered, in an effort to provide options that would 
allow more or less expansion depending on projected resource availability. Table 3-4 shows the 
MSAs grouped by the nine census divisions and ranked within division by their 1999 population. 
The columns then show which specific MSAs are included in the current design (21 MSAs); 
which MSAs would be included in each of three rank-by-population scenarios (the largest 36, 

45, and 54 MSAs overall); and which MSAs would be included in each of three rank-within- 
census division scenarios (the largest 4, 5, or 6 MSAs in each of the nine Census Divisions). 
Because sufficient local interest in DAWN was shown in the current 21 MSAs, each of the 
options shown in Table 3-4 “protects” the 21 MSAs - that is, the options are included in all 
scenarios whether or not they meet the specific population ranking requirements of the overall 
approach. 

The various combinations of metropolitan areas shown in Table 3-4 were then plotted on 
maps that were color-coded to correspond to each of the columns of the table. This exercise 
proved to be extremely valuable, because the maps provided a clear visual reference on which 
to base a final decision. Specifically, the maps illustrated that an expansion plan that added 
MSAs in descending order of population would, for the most part, add MSAs to regions where 
there was already significant coverage (typically the Pacific and South Atlantic divisions). On 
the other hand, expanding to include some number of MSAs in each census division ensured 
the inclusion of many of the most populous areas, along with some coverage in every major 
geographic area. Of particular importance was obtaining coverage of the East South Central 
division (which presently has no representation in DAWN), as well as enhancing coverage in the 
New England division and the rest of the central United States. 

The decision was made to adopt an expansion strategy that would add MSAs by population 
within census division, while “protecting” each of the current 21 MSAs. The redesign team was 
instructed to proceed under the assumption that sufficient resources would be available to 
roughly double the number of MSAs currently represented in DAWN. The proposed expansion 
approach, shown in the next-to-last column in Table 3-4, calls for a total of 48 MSAs - the five 
largest in each of the nine census divisions (based on 1999 Census data), plus any of the 21 
current MSAs falling outside the top 5 in their division. Methods used to arrive at this particular 
design decision are described in more detail here.. 
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Table 3-4. Geographic Coverage Options Considered in DAWN Sample Redesign 



Proposed 

Design 


CENSUS DIVISION & METROPOLITAN AREA 

Current 21 

MSAs 

(21) 

36 Largest 

MSAs+ 

(38) 

45 Largest 

MSAs+ 

(46) 

54 Largest 

MSAs+ 

(54) 

4 Largest 

MSAs in 9 

Divisions* 

(42) 

5 Largest 

MSAs in 9 

Divisions* 

(48) 

6 Largest 

MSAs in 9 

Divisions* 

(56) 

NEW ENGLAND 




Boston-Worcester-Lawrence-Lowell-Brockton, MA-NH NECMA 





V 

S 

V 

New Haven-Bridgeport-Stamford-Waterbury-Danbury, CT NECMA 

S 



S 

S 

V 

Hartford, CT NECMA 





S 

S 

V 

Providence-Warwick-Pawtucket, Rl NECMA 





V 

S 

V 

Springfield, MA NECMA 






S 

V 

Portland, ME NECMA 







V 

MIDDLE ATLANTIC 




New York, NY PMSA 


s 



V 

V 

V 

Philadelphia, PA-NJ PMSA 

S 

s 



V 

s 

V 

Nassau-Suffolk, NY PMSA 


s 



V 

s 

V 

Pittsburgh, PA MSA 


V 



V 

s 

S 

Newark, NJ PMSA 

S 

s 



V 

s 

V 

Bergen-Passaic, NJ PMSA 







V 

Buffalo-Niagara Falls, NY MSA 





S 

s 

V 

Middlesex - Somerset - Hunterdon, NJ PMSA 








Monmouth - Ocean, NJ PMSA 








EAST NORTH CENTRAL 




Chicago, IL PMSA 


s 



V 

✓ 

S 

Detroit, Ml PMSA 


s 



V 

s 

V 

Cleveland-Lorain-Elyria, OH PMSA 


s 



V 

s 

V 

Cincinnati, OH-KY-IN PMSA 


s 



V 

s 

V 

Indianapolis, IN MSA 






s 

V 

Columbus, OH MSA 







V 

Milwaukee-Waukesha, Wl PMSA 
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Table 3-4. Geographic Coverage Options Considered in DAWN Sample Redesign (continued) 



Proposed 

Design 


CENSUS DIVISION & METROPOLITAN AREA 

Current 21 

MSAs 

(21) 

36 Largest 

MSAs+ 

(38) 

45 Largest 

MSAs+ 

(46) 

54 Largest 

MSAs+ 

(54) 

4 Largest 

MSAs in 9 

Divisions+ 

(42) 

5 Largest 

MSAs in 9 

Divisions+ 

(48) 

6 Largest 

MSAs in 9 

Divisions* 

(56) 


WEST NORTH CENTRAL 


Minneapolis-St. Paul, MN-WI MSA 

v' 

y 

y 

y 

y 

y 

y 

St. Louis, MO-IL MSA 

y 

y 

y 

y 

y 

y 

y 

Kansas City, MO-KS MSA 


y 

y 

y 

y 

y 

y 

Omaha, NE-IA MSA 





y 

y 

y 

Wichita, KS MSA 






y 

y 

Des Moines, IA MSA 







y 

SOUTH ATLANTIC 




Washington, DC-MD-VA-WV PMSA 

V 

y 

y 

y 

y 

y 

y 

Atlanta, GA MSA 

y 

y 

y 

y 

y 

y 

y 

Baltimore, MD PMSA 

V 

y 

y 

y 

y 

y 

y 

Tampa-St. Petersburg-Clearwater, FL MSA 


y 

y 

y 

y 

y 

y 

Miami, FL PMSA 

y 

y 

y 

y 

y 

y 

y 

Norfolk-Virginia Beach-Newport News, VA-NC MSA 


y 

y 

y 



y 

Fort Lauderdale, FL PMSA 



y 

y 




Orlando, FL MSA 



y 

y 




Charlotte-Gastonia-Rock Hill, NC-SC MSA 



y 

y 




Greensboro - Winston-Salem - High Point, NC MSA 




y 




Raleigh - Durham - Chapel Hill, NC MSA 




y 




EAST SOUTH CENTRAL 




Nashville, TN MSA 




y 

y 

y 

y 

Louisville, KY-IN MSA 





y 

y 

y 

Birmingham, AL MSA 





y 

y 

y 

Knoxville, TN MSA 





y 

y 

y 

Mobile, AL MSA 






y 

y 

Johnson City - Kingsport - Bristol, TN-VA MSA 







y 


"vl 

GO 
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Table 3-4. Geographic Coverage Options Considered in DAWN Sample Redesign (continued) _ 

Proposed 

Design 


CENSUS DIVISION & METROPOLITAN AREA 

Current 21 

MSAs 

(21) 

36 Largest 

MSAs+ 

(38) 

45 Largest 

MSAs+ 

(46) 

54 Largest 

MSAs+ 

(54) 

4 Largest 

MSAs in 9 

Divisions* 

(42) 

5 Largest 

MSAs in 9 

Divisions* 

(48) 

6 Largest 

MSAs in 9 

Divisions* 

(56) 

WEST SOUTH CENTRAL 

Houston, TX PMSA 


S 




S 


Dallas, TX PMSA 


S 



V 

S 

V 

Fort Worth-Arlington, TX PMSA 


S 




S 

V 

San Antonio, TX MSA 


V 

S 



S 


New Orleans, LA MSA 


S 




S 


Austin - San Marcos, TX MSA 








MOUNTAIN 




Phoenix-Mesa, AZ MSA 






S 


Denver, CO PMSA 






S 


Las Vegas, NV-AZ MSA 






S 


Salt Lake City-Ogden, UT MSA 





V 

S 

V 

Tucson, AZ MSA 






S 


Albuquerque, NM MSA 







V 

PACIFIC 




Los Angeles-Long Beach, CA PMSA 


S 



V 

S 


Riverside-San Bernardino, CA PMSA 


V 




S 


San Diego, CA MSA 

V 

V 




✓ 


Orange County, CA PMSA 


S 

S 




V 

Oakland, CA PMSA 


S 




S 

V 

Seattle-Bellevue-Everett, WA PMSA 

V 

S 

S 



S 


Portland-Vancouver, OR-WA PMSA 


S 






San Francisco, CA PMSA 


S 




S 


San Jose, CA PMSA 


s 






Sacramento, CA PMSA 


s 





































































































MODELING ED POPULATION DATA 


To assign facilities to strata, it was necessary to model population data - that is, to make 
some determination about the size of each eligible facility, as expressed not only in total ED 
visits but also in the number of heroin, cocaine, marijuana, and methamphetamine episodes 
seen in each facility. While total ED visit data are available for all eligible facilities from 
secondary sources, DAWN is the only source of information on drug-related episodes. Thus, 
modeling was required to estimate the total number of drug-related and drug-specific episodes 
for facilities not currently participating in DAWN. 

To accomplish this, each eligible record (facility) on the DAWN frame was assigned a 
reported annual ED visits count, which was based on the DAWN and American Hospital 
Association (AHA) survey data and consistent with the methodology used for DAWN benchmark 
adjustment. Each eligible record was also assigned an estimate of the number of DAWN 
episodes for the four major drug groups (marijuana, cocaine, heroin, and methamphetamine) 
and in total. These annual estimates were modeled (for eligible units not in the DAWN sample 
or not responding) based on the relationship identified between these variables and annual ED 
visits, within region and ownership, as follows: 

Xhijk~MAX(Q,BQInj + B\lnj Vk ) 


Where 


Xhijk = the annual estimate of episodes for region h, ownership class i, drug j, unit k 
BOhij = the model intercept term for region h, ownership class i, drug j 
B1 hij = the model slope parameter for region h, ownership class i, drug j 
Vk = annual ED visits for unit k. 

SAS PROC REG was used to obtain the model parameters. 

These estimated annual counts were the basis for the final stage of the multivariate sample 
design optimization. It was important in this regard that these estimates be calculated as the 
predicted value plus or minus some standardized residual. Westat’s WESDECK hot-deck 
imputation software was used to select donors that would be used to add or subtract a 
standardized residual to the predicted annual estimate of episodes given above. (Although 
WESDECK is Westat’s proprietary software, hot-deck imputation is a well-established statistical 
procedure.) 

For each eligible unit on the frame and not reporting, a modified annual estimate of 
episodes was calculated as follows: 


y hijk MAX 


f 

0 ’ X hijk 
V 


y hijD(k) 
XhijD(k) 




J 


Where 
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Y’hijk 

Xhijk 

YhijD(k) 

XhijD(k) 


the final estimated annual episodes for region h, ownership class i, drug j, and 
unit k 

the model estimated annual episodes for region h, ownership class i, drug j, 
unit k 

the reported annual episodes for region h, ownership class i, drug j, and donor 
for k 

the model estimated annual episodes for region h, ownership class i, drug j, 
donor for k 


A note is in order regarding the above methodology. The relationship between ED visits and 
drug-specific episodes is not especially strong, even within region and ownership. Models were 
somewhat imprecise and as a result contributed variance to the estimated drug-specific 
episodes. It may be possible, given additional resources, to obtain more precise models that in 
turn would result in smaller required sample sizes. This is uncertain, however, given the 
variability observed in the DAWN data and the fact that the process had already made use of 
the most significant quantitative variable (ED visits) available for the entire population. The 
amount of reduction in sample sizes would be directly related to the success of the modeling 
and was therefore difficult to speculate. The improved modeling task also competed with other 
sample redesign tasks. Concerns such as these, along with competition for resources among 
redesign activities, SAMHSA’s experience with modeling state-level estimates for the NHSDA, 
and their resulting concerns about modeling at the hospital level for DAWN led to the decision 
not to develop these models further. 


SAMPLE DESIGN OPTIMIZATION PROCESS 

Sample design optimization is trivial in the univariate case, in that whether calculating the 
sample size required for specified precision or calculating the precision given by a specific 
sample size, one simply solves for one unknown in an algebraic equation. For example, under 
simple random sampling, the precision obtained by a sample of size n is calculated as follows: 

x.SRS n N n 


Where 




i =1 


N -1 


/= 


n 
1 V 
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Similarly, under stratified random sampling, an explicit solution for the optimal allocation of 
n sampling units to h strata is available as follows: 


nh~n — 

I 


In the multivariate case, however, no single equation is either available or appropriate. In 
this case, a mathematical programming approach is required. 

Mathematical programming problems have the following three characteristics: 

■ Decision variables - Variables allowed to take on any values, subject to certain 
constraints 

■ Objective function - A function of the decision variables to be maximized or minimized 

■ Constraints - Explicit constraints on the decision variables themselves or implicit 
constraints on other functions of the decision variables 

In general, the decision variables are quantities that are allowed to move about, as needed, 
in order to solve all other aspects of the mathematical programming problem. Sometimes the 
decision variables are subject to certain reality or practicality-driven constraints. For example, in 
manufacturing, the decision variables could be the number of units of a particular product to 
produce. The objective function is a function of the decision variables and can be either linear 
on nonlinear. The nature of the problem often requires that the objective function be maximized, 
minimized, or set to a particular value. For example, in manufacturing, the objective function 
may be to maximize the profit from producing a given lot of units. The constraints are often in 
terms of values that the decision variables must respect but can also be another function of the 
decision variables. For example, in the manufacturing problem negative numbers of units would 
not be allowed, there may be a minimum number of units of each product that must be 
produced, and the common resources consumed may be a function of the number of units of 
more than one product and must be kept at or below a particular level. The particular 
mathematical programming problems for the DAWN sample redesign are presented below. 

SAS PROC NLP, part of the SAS OR library, was used to solve these problems with the Quasi- 
Newton method. 
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MULTIVARIATE SAMPLE DESIGN OPTIMIZATION 


The DAWN sample design optimization problem (the sample size minimization version) can 
be expressed in the following mathematical programming notation: 

Minimize: 


L Hi 

T= E L nij 

i =1 7=1 


Subject to: 

1) 2 <nij<Nij 

2) RSEi{x k )<Ti(x k ) 

Where: 


i 

= the 

j 

= the 

L 

= the 

Hi 

= the 

nij 

= the 

Nij 

= the 

RSEiixk ) 

= the 

Tiixk ) 

= the 


primary stratum index (e.g., MSA), 
secondary stratum index (e.g., ownership X size), 
number of primary strata (# of specifically targeted MSAs + 1) 
number of secondary strata in primary stratum i 

sample size in stratum ij (these are the decision variables) 

population size in stratum ij 

relative standard error of estimate k in stratum i 

target relative standard error of estimate k in stratum i 


and the relative standard error is expressed as follows: 


RSEiixk)- 



for MSA estimates and 


RSE N (x k ) 


I L Hi jy. 2 s 2 (\ - 
I y y yy il v 1 Jill 

i=lj=l 


n,-, 


x Nk 


for the national estimates. 


OVERALL SAMPLE DESIGN OPTIMIZATION PROCESS 

The sample design optimization process developed for DAWN is shown in the flow chart in 
Figure 3-1. The sample design optimization process consisted of the following steps: 

■ Determine primary stratification based on specified MSAs. 
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Figure 3-1. Dawn Sample Design Scenarios Evaluation System 
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■ Determine secondary stratification based on other sample design variables (e.g., 
ownership, size.) 

■ Model and calculate population values from AHA and DAWN data, by sampling 
stratum. 

■ Solve for minimum sample size required to satisfy all constraints. 

■ Record and report scenario and results. 

■ Provide results for SAMHSA’s review. 

The primary stratification was determined based on the MSAs targeted in a specific 
scenario. Secondary stratification was determined based on the other stratification variables 
used in a specific scenario. For example, stratification by hospital ownership (public, non-public) 
and size (based on annual ED visits) were used and identified as being considerably more 
effective than stratification by MSA or MSA and ownership alone. Up to four size-specific strata 
were formed within each MSA and ownership category. The stratification by size depended on 
the number of sampling units available within each MSA and ownership category, as shown in 
Table 3-5. 


Table 3-5. Stratification by Size 


Number of Sampling Units 

Available Within MSA 

Number of Size Strata 

0...3 

1 

4 

2 

5 

2 

6 

2 

7 

2 

8 or more 

4 


The sampling units were sorted in descending order of the annual measure of ED visits and 
split into strata with an even number of sampling units. When this was not possible the odd 
number of units was simply assigned to the latter strata. 

Given the stratification and modeled and calculated population values described above, 
SAS PROC NLP was used to solve for the minimum responding sample size required for a 
given sample design scenario to satisfy all constraints, as expressed above. These results were 
recorded in a scenario log, which is the basis for all tables included in this report. 
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SAMPLE DESIGN SCENARIOS AND RESULTS 


One of DAWN’S greatest analytical strengths is its ability to say something about individual, 
local areas. The national estimate available from DAWN is not the system’s primary analytic 
objective, but it is useful for putting individual MSA estimates in perspective, especially changes 
in the trends of such estimates. The MSA analytic objective is the main driving factor behind the 
minimum required responding sample sizes discussed here. It is therefore useful to compare 
the sample sizes for scenarios requiring MSA estimates to the sample sizes that would be 
required to produce a national estimate exclusively. This, in turn, helps put in perspective the 
sample sizes that are required within the national panel to yield a national estimate with 
specified precision, given the sample sizes allocated to the targeted MSAs. 


NATIONAL ESTIMATE-ONLY SCENARIOS 

Table 3-6 provides the sample sizes required for various scenarios involving a national 
estimate only. For example, 91 sampled units are required to produce a national estimate of ED 
visits with an RSE of 10 percent under a simple random sample design. By contrast, 202 
sampled units are required to produce a national estimate of total episodes with an RSE of 10 
percent under a stratified random sample design, where the stratification was by ownership and 
size. Expanding that scenario to include estimates for cocaine episodes, heroin episodes, and 
marijuana episodes with RSEs of 10 percent leads to a required sample size of 544. Finally, 

660 sampled units are required to produce a national estimate of total episodes with an RSE of 
5 percent. 


Table 3-6. National Estimate-Only Scenarios _ 

Scenario Sample Size 

Optimized, SRS, 10% RSE on ED visits 91 

Optimized, Stratified SRS, 10% RSE on total episodes 202 

Optimized, Stratified SRS, 10% RSE on total episodes, 544 

Cocaine, Fleroin, Marijuana 

Optimized, Stratified SRS, 5% RSE on total episodes 660 
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EARLY ROUNDS OF SCENARIOS 


As described above, a sample design scenario represents the combination of the level at 
which estimates are required, the specific estimates that are required, and the precision level 
that is required for those estimates. The sample design scenarios that were run through the 
optimization process included the following choices for each of these parameter classes: 


Level: 


1) National and the current 21 DAWN MSAs 

2) National and the top 21 MSAs in 1999 

3) National and top 27 MSAs in 1999 

4) National and the top 50 MSAs in 1999 


Estimates: 1) 

2 ) 

3) 

4) 

5) 


Total annual ED visits 

Total annual Marijuana/Hashish episodes 

Total annual Cocaine episodes 

Total annual Heroin/Morphine episodes 

Total annual Methamphetamine/Speed episodes 


Precision: 1) 

2 ) 


RSEs = 5% 
RSEs= 10% 


The scenario results obtained were in line with the redesign team’s expectations and the 
current DAWN sample design. Differences between the results are direct consequences of 
changes in targeted areas (MSAs), the number of estimates included in the optimization (ED 
visits, episodes for the 4 top drug categories), the relative standard error constraints applied 
(RSEs), and whether RSE constraints were applied specifically to the national panel. The 
results range from a minimum required sample size of 91 hospitals for a national estimate of ED 
visits only with an RSE <= 10%, to a maximum required sample size of 2,386 hospitals for 
estimates for the nation as well as each of the top 50 MSAs with RSEs <= 5% for all five 
estimates (ED visits and the 4 drug categories). The results are described in more detail below. 


Table 3-7 shows a select few sample scenarios and the responding sample sizes needed to 
achieve either 10 percent or 5 percent RSEs on the desired estimates. 


Table 3-7. Actual or Minimum Required Responding Sample Size by Design and MSA 
Precision Level 


Design 

RSE = 10% 

RSE = 5% 

Current (21 MSAs, ED visits and total episodes) 

488 

n/a 

Optimized (21 MSAs, ED visits and total episodes) 

459 

609 

Optimized (21 MSAs, 5 estimates) 

899 

1,614 

Optimized (Top 27 MSAs, 5 estimates) 

928 

1,591 

Optimized (48 MSAs, 4 estimates) 

950 

n/a 

Optimized (Top 50 MSAs, 5 estimates) 

1,148 

1,723 
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The first and fifth rows of this table represent the current design and the proposed new 
design, respectively. The current sample design obtains reasonably precise national and MSA 
estimates of ED visits and total drug-related episodes with a responding sample size of 
approximately 488 sampled units. The current RSEs for the national estimates are around 10 
percent. The results by MSA and specific drug vary under the current design, with the RSEs of 
estimates for some MSAs and some specific drugs (e.g., methamphetamine) being far in excess 
of 10 percent (discussed in detail below). 

The minimum required responding sample sizes obtained from the sample design 
optimization process ranged from 459 sampled units for a minimum MSA RSE of 10 percent on 
ED visits and total drug-related episodes to 1,723 sampled units for a minimum MSA RSE of 5 
percent on ED visits and cocaine, heroin, marijuana, and methamphetamine episodes. Note that 
the first and second rows of the table are not exactly comparable. This is due to the fact that the 
actual achieved MSA RSEs for the current design vary considerably, while the expected MSA 
RSEs for the optimized design are at or below the given RSE level for all MSAs. 

The minimum required sample sizes increase as additional estimates or additional MSAs 
are added to the sample design scenarios. For example, the third row of Table 3-7 shows the 
minimum required sample sizes for minimum RSEs of 10 percent or 5 percent in the current 21 
MSAs for each of five estimates of interest (ED visits, cocaine episodes, heroin episodes, 
marijuana episodes, methamphetamine episodes). The fifth row shows the minimum required 
sample sizes for the proposed design, with minimum MSA RSEs of 10 percent for four 
estimates of interest (total drug-related episodes, cocaine episodes, heroin episodes, and 
marijuana episodes). The evolution and selection of these combinations of requirements are 
detailed in the sections that follow. 

Next, Table 3-8 shows the minimum sample sizes required by stratum for several different 
sets of estimates within five selected MSAs. This table is illustrative of the output generated for 
each different list of candidate MSAs given desired estimates and precision constraints. The 
estimates shown here include ED visits only, ED visits and all four major drug categories, ED 
visits and three of the major drug categories (less methamphetamine), and each of the four 
major drug categories separately. Table 3-8 shows RSEs =10 percent. Similar tables were 
produced for RSEs = 5%. The purpose of this table is to show the differences in minimum 
required sample sizes, at the finest stratification level, for various subsets of estimates. Joint 
review of a considerable number of such tables by the redesign team and OAS informed the 
final decisions regarding the scope and composition of the proposed new sample design. 
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Table 3-8. Minimum Sample Sizes Required by Stratum and Estimates, Selected MSAs 


Stratified by MSA, ownership, and size. RSE constraint =10%. 


Metropolitan Area 

Variables 

Ownership = Public 

Ownership = Private 

TOTALS 

Size group 

Size group 

1 

2 

3 

4 

5 

6 

7 

8 

LOS ANGELES-LONG BEACH, CA MSA 

Strata population count 

3 

3 

0 

0 

20 

20 

20 

21 

87 

ED visits 

2 

2 



2 

2 

2 

2 

12.00 

ED visits & 4 Drugs 

3 

2.89 



8.15 

11.05 

5.1 

2 

32.19 

ED visits & 3 Drugs 

3 

3 



5.91 

4.56 

5.94 

2 

24.41 

Cocaine 

3 

3 



5.91 

4.56 

5.94 

2 

24.41 

Heroin 

3 

2.49 



5.5 

4.34 

2.73 

2 


Marijuana/hashish 

3 

3 



5.18 

3.7 

3.82 

2 


Methamphetamine/speed 

3 

2.44 



8.27 

11.23 

5.18 

2 

32.12 

NEW YORK, NY MSA 

Strata population count 

3 

3 

3 

4 

16 

16 

16 

17 

78 

ED visits 

2 

2 

2 

2 

2 

2 

2 

2 

16.00 

ED visits & 4 Drugs 

2 

2 

2 

2.46 

11.2 

16 

8.55 

7.72 

51.93 

ED visits & 3 Drugs 

2 

2 

2 

3.15 

9.7 

16 

3.75 

9.31 

47.91 

Cocaine 

2 

2 

2 

3.69 

7.33 

16 

4.21 

9.92 

47.15 

Heroin 

2 

2 

2 

2 

11.92 

16 

2.64 

7.66 

46.22 

Marijuana/hashish 

2 

2 

2 

2.63 

5.09 

16 

5.19 

5.35 

40.26 

Methamphetamine/speed 

2 

2 

2 

2 

11.89 

16 

9.54 

4.71 

50.14 

CHICAGO, IL MSA 

Strata population count 

3 

0 

0 

0 

20 

21 

21 

21 

86 

ED visits 

2 




2.8 

2 

2 

2 

10.80 

ED visits & 4 Drugs 

3 




20 

16.17 

14.52 

4.07 

57.76 

ED visits & 3 Drugs 

3 




20 

16.21 

8.17 

6.34 

53.72 

Cocaine 

3 




20 

13.81 

9.11 

7 

52.92 

Heroin 

3 




14.53 

21 

3.11 

2.25 

43.89 

Marijuana/hashish 

2.27 




20 

12.43 

9.77 

7.61 

52.08 

Methamphetamine/speed 

3 




20 

4.7 

17.39 

3.96 

49.05 

PHILADELPHIA, PA-NJ MSA 

Strata population count 

0 

0 

0 

0 

15 

16 

16 

16 

63 

ED visits 





2.2 

2 

2 

2 

8.20 

ED visits & 4 Drugs 





15 

5.78 

11.65 

6.35 

38.78 

ED visits & 3 Drugs 





15 

3.94 

2.9 

8.4 

30.24 

Cocaine 





15 

3.62 

2.73 

8.83 

30.18 

Heroin 





15 

5.53 

2 

4.3 

26.83 

Marijuana/hashish 





15 

3.87 

3.47 

2 

24.34 

Methamphetamine/speed 





15 

5.96 

12.9 

3.52 

37.38 

WASHINGTON, DC-MD-VA-WV MSA 

Strata population count 

1 

0 

0 

0 

9 

9 

9 

10 

38 

ED visits 

1 




2 

2 

2 

2 

9.00 

ED visits & 4 Drugs 

1 




8.88 

9 

7.24 

2.35 

28.47 

ED visits & 3 Drugs 

1 




9 

4.91 

4.8 

4.07 

23.78 

Cocaine 

1 




9 

4.55 

3.16 

3.72 

21.43 

Heroin 

1 




6.84 

4.8 

2 

2 

16.64 

Marijuana/hashish 

1 




9 

4.91 

4.8 

4.07 

23.78 

Methamphetamine/speed 

1 




5.17 

9 

8.92 

2.44 

26.53 






























































































































































































































































































LAST ROUND OF SCENARIOS 


The last round of optimization scenarios focused on four different lists of MSAs and three 
different combinations of national and MSA-specific RSE constraints. The four different lists of 
MSAs were as follows: 

1. Top 36 MSAs by population, plus any of current 21 otherwise excluded (38 total) 

2. Top 4 MSAs in each of 9 census divisions, plus any of the current 21 otherwise excluded 
(42 total) 

3. Top 45 MSAs by population, plus any of current 21 otherwise excluded (46 total) 

4. Top 5 MSAs in each of 9 census divisions, plus any of the current 21 otherwise excluded 
(48 total) 

The three different combinations of national and MSA-specific RSE constraints were as 
follows: 

1. Twenty percent RSEs for national estimates, 10 percent RSEs for MSA estimates 

2. Fifteen percent RSEs for national estimates, 10 percent RSEs for MSA estimates 

3. Ten percent RSEs for national estimates, 10 percent RSEs for MSA estimates 


The estimated minimum required responding sample sizes for these different scenarios are 
shown in Table 3-9. 


Table 3-9. Estimated Minimum Required Responding Sample Sizes for Last Round of 
_ Scenarios (includes national panel sample size) _ 


Scenario 


RSE Constraints National/MSA 

20% /10% 

15%/10% 

10%/10% 

Top 36+ (38 total) 

825 

868 

980 

Top 4 in 9+ (42 total) 

861 

901 

1,005 

Top 45+ (46 total) 

909 

946 

1,046 

Top 5 in 9+ (48 total) 

911 

950 

1,051 
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The sample size required in the national panel for these different scenarios are shown in 
Table 3-10. 


Table 3-10. Estimated National Panel Sample Sizes for Last Round of Scenarios 


Scenario 


RSE Constraints National/MSA 


20% /10% 

15%/10% 

10%/10% 

Top 36+ (38 total) 

64 

107 

218 

Top 4 in 9+ (42 total) 

60 

100 

204 

Top 45+ (46 total) 

57 

95 

194 

Top 5 in 9+ (48 total) 

59 

97 

199 


It was through review of these estimated required responding sample sizes, including the 
national panel sample sizes, and the actual expected RSEs obtained through these sample sizes 
that the proposed new design was decided upon. The proposed new design includes the top 5 
MSAs in each of the 9 census divisions, plus any of the current 21 MSAs otherwise excluded (48 
MSAs total), with national estimate RSE constraints of 15 percent and MSA estimate RSE 
constraints of 10 percent. This design entails an estimated sample size of 950 responding units, of 
which 97 are allocated to the national panel. It was felt that this design best balanced the desire to 
expand the sample’s geographic coverage and improve the precision of the estimates obtained 
while considering the resources needed to recruit, retain, and manage a data system of this size. 

As has been discussed earlier, the actual expected RSE will be less than or equal to the RSE 
constraints. Table 3-11 presents the actual expected RSEs for the proposed new design. 


Table 3-11. Actual Expected RSEs for the Proposed New Design 


Estimate 

National 

estimate 

Mean for 48 MSA 
estimates 

MAX for MSA estimates 

Total episodes 

9.0% 

3.9% 

10.0% 

Cocaine episodes 

13.0% 

6.3% 

10.0% 

Heroin episodes 

15.0% 

7.2% 

10.0% 

Marijuana episodes 

15.0% 

7.8% 

10.0% 


The RSEs in Table 3-11 compare favorably to the RSEs achieved by the current design (see 
Table 3-3). The mean MSA RSEs for the proposed new design are roughly half those of the MSAs 
included in the current design. The maximum RSEs are considerably lower for the new design. The 
national estimate RSEs for the new design are slightly higher than those of the current design. 
Increasing the national panel (and hence total) sample size by 100 responding units would bring 
the national estimate RSEs for the new design to or below the level of the current design. 
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COST-BASED OPTIMIZATION 


All of the scenarios and optimization runs described above involved the relatively naive 
assumption that all sampling units were of equal cost. OAS used the estimated minimum required 
responding sample sizes as a proxy for cost, relative to the current responding sample size and 
costs. However, it may cost more or less to recruit one facility relative to another, to retain that 
facility, and to compensate that facility for participation in DAWN. Moreover, actual costs may be 
different under the new design if new recruitment, retention, and/or reimbursement strategies are 
developed. A fully-informed cost-based optimization for the new implementation would require that 
the direct reimbursement model be known for the new design and implementation. However, the 
redesign team was not tasked with developing that new reimbursement model; this task was 
explicitly assigned to the data collection contractor that would be responsible for implementing the 
new sample. Therefore, for the purpose of running a final cost-based optimization to compare to 
the naive model, the current reimbursement model was used. 


Current Reimbursement Model. The current direct reimbursement model is as follows: 

0/ - f {Bq] + + Bujyj ) z ij 


Where 


xij = the mean annual ED visits within stratum ij (i=MSA/national panel, 
j=ownership/size stratum) 

v„ = the mean annual total episodes within stratum ij 

y 

Boij = the mean minimum compensation level within stratum ij 

B\ij = the mean ED visits coefficient from the current cost model within stratum ij 

B 2 ij = the mean Total Episodes coefficient from the current cost model within stratum ij 

F = 0.90 (this is the factor that gives SAMHSA room to negotiate) 

Z = the cost of living adjustment (COLA) from the current cost model, for primary 
stratum i 

The direct reimbursement model coefficients varied by whether a unit was located in a central 
city or not, as follows in Table 3-12: 

Table 3-12. Direct Reimbursement Model Coefficients by Fa cility Location. 


Coefficient 

In a Central City 

Not in a central city 

BO 

45.0 

313.7 

B1 

0.066 

0.035 

B2 

1.879 

1.735 


Cost-Minimization Run. The sample design optimization problem was changed from one 
that minimizes sample size to one that minimizes cost. The cost was limited to the current 
reimbursement protocol. Another optimization problem was run for the proposed design only. The 
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intention was to compare the sample allocation results to those previously obtained under the 
simpler sample size minimization problem. 

A cost minimization version of the sample design optimization problem can be stated as 
follows: 

Minimize: 


L Hi 

t= L L cijnij 

i =1 j =1 


Subject to: 


1 ) 2< nij < Nij 
2 ) RSEi(x k )<Ti(x k ) 

Where: 


i 

= the 

j 

= the 

L 

= the 

Hi 

= the 

c ii 

= the 

ny 

= the 

Nij 

= the 

RSEiixk ) 

= the 

Tiixk ) 

= the 


primary stratum index (e.g., MSA). 

secondary stratum index (e.g., ownership X size). 

number of primary strata (# of specifically targeted MSAs + 1) 

number of secondary strata in primary stratum i 

average cost of sampling a unit in stratum ij (see below) 

sample size in stratum ij (these are the decision variables) 

population size in stratum ij 

relative standard error of estimate k in stratum i 

target relative standard error of estimate k in stratum i 


and the relative standard error is expressed as follows: 



x ik 


RSE N (x k ) 


jhk i 


x Nk 
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Cost Data. Since the coefficients for the direct reimbursement model were not constant within 
the strata proposed for the new sample design, an average annual direct reimbursement cost for 
each unit within a particular stratum was calculated as follows: 

Cij = ^ +B\ ijXij +B2ijyij)zij 

Where 


Xij 

= the mean 

y& 

= the mean 

B 0ij 

= the mean 

B\ij 

= the mean 

B2ij 

= the mean 


annual ED visits within stratum ij 

annual total episodes within stratum ij 

minimum compensation level within stratum ij 

ED visits coefficient from the current cost model within stratum ij 

Total Episodes coefficient from the current cost model within stratum 


ij 


Evaluation. Only very minor differences in the sample allocation and estimated 
reimbursement costs were found between the sample size minimization and cost minimization 
runs. The differences in required responding sample size and estimated direct reimbursement 
were as shown in Table 3-13. 


Table 3-13. Cost-Minimization Results 


Optimization 

problem 

Required responding sample size 

Estimated direct reimbursement 

Minimize sample 

949 

$3,361,490.41 

size 



Minimize cost 

961 

$3,334,997.53 


Comparison of the sample size-minimization results with the cost-minimization results yielded 
the following observations: 

1. The difference in sample size between the two approaches is a total of 12 units. 

2. The difference in estimated cost between the two approaches is a total of $26,492.88. 

3. The COLA term does not affect the allocation. Identical results were obtained using 
uniform COLAs of 1.0 compared with the COLAs obtained from the 2000 Statistical 
Abstract of the United States. 

4. The allocation at the MSA level changed only very slightly between the sample size- 
minimization and cost-minimization approaches. 

5. The allocation across strata within a given MSA changed slightly between the sample 
size-minimization and cost-minimization approaches, indicating that the direct 
reimbursement model coefficients are having only a minor effect on the allocation. 
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These observations may or may not hold under a different cost or direct reimbursement model. 
It is likely that these observations will hold under any similar models, as the driving factor at the 
MSA level is the MSA precision constraint, whenever these constraints are more stringent than 
what is required at the national level. Under other sample design circumstances, cost-minimization 
results can be considerably different from sample size-minimization results. 


OTHER SAMPLING ISSUES 

This section lists a number of statistical issues that were considered, at various levels of detail, 
in addition to the DAWN sample redesign analysis. Some of these issues have been discussed in 
the past, some were discussed in DAWN management meetings, and some are appropriate to 
consider now given the DAWN redesign in general. Note that resolution of many of these issues 
was beyond the scope of the redesign contract and must be handled by the analytic contractor as 
the new design is implemented. The purpose of this section is to document the issues that were 
considered and to identify areas where specific approaches will need to be developed. 

These issues are presented in the likely order of implementation. For example, issues such as 
overlap control and the merger/demerger protocol should be resolved prior to drawing a new 
sample. Issues such as adjustments for missing data and variance estimation should be resolved 
before estimates are required from the new sample. Other issues, such as sample maintenance 
and panel rotation and phasing-in strategies are ongoing issues. 


Overlap Control 

The redesign team recommended that overlap control be considered as part of the DAWN 
sample redesign activities and researched this option. Overlap control is often considered as a 
recruitment cost-reduction technique when an ongoing study implements a new sample design but 
continues to use the same kind of sampling units (e.g., hospitals). Versatile techniques exist that 
can maximize the expected overlap between the old and new samples while maintaining the 
desired statistical features of the new sample design. These techniques become increasingly 
complicated as the measures of size, stratification, and definition of sampling units change from 
the old sample design to the new sample design. 

This section presents the results of the redesign team’s research and the resulting 
recommendation. The research indicated that overlap control can not be implemented exactly (i.e., 
optimally) due to the size of the overlap control problem. There are also cases of births, deaths, 
mergers, and demergers in which the concept of overlap may be difficult to define. There are 
techniques for approximate overlap control that could be applied in the problematic strata. 3 Given 
the challenges faced, it was recommended that overlap control be implemented only in the national 
panel in the new DAWN sample design. If overlap control is desired in other strata, careful study of 
those strata is recommended to ensure that the new sample is properly implemented. 

In order to perform overlap control correctly, it is necessary to know the joint probabilities of 
selection from any previous rounds of overlap control. Usually, strata are selected independently, 
but if an overlap control selection method is used, the independence of strata assumption can be 
violated. Because the current design (developed in 1988) reported using overlap control 


3 See, for example, (1999). The maximization and minimization of sample overlap problems: A half century of results: or Chowdhury, 
Chu, and Kaufman. (2000). Minimizing overlap in NCES surveys, 2000 ASA Proceedings of JSM, Section on Survey Research. 
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procedures with a previous design (i.e., the 1970’s design), it is necessary to obtain the joint 
probabilities of the 1970’s - 1988 design if overlap control procedures are to be used in selecting 
the new DAWN sample. 

SAMHSA provided historical documents on their methods of sample selection from the 
previous design. SAMHSA identified the 1988 internal document “Development and 
Implementation of a Probability Sample Design for the Drug Abuse Warning Network” as the 
authoritative account to be followed. This document includes a section that describes the overlap 
control that was performed as part of the 1988 design. Overlap control was implemented in only 
three areas and via the Keyfitz procedure: New York, Chicago, and in strata 8 and 9 of the national 
panel. In all other areas and strata, the selections in the 1988 sample were independent of the 
1970’s design. If overlap control is performed again in any of the strata in New York, Chicago, or 
the national panel, then the previous overlap control must be taken into account. 

It was recommended that if SAMHSA desires to employ overlap control between the proposed 
sample design and the 1988 sample, then the conditional probabilities used in the 1988 overlap 
control must be used as part of the new overlap control. It appears that in most DAWN areas, no 
overlap control was performed in the 1988 sample. Exact or approximate overlap control is 
therefore feasible for the proposed new design. The benefit of this overlap will depend on the 
number of units in common between the proposed new design and the 1988 sample, as well as the 
differences in the relevant recruitment costs. Due to the large number of sampled units within each 
targeted MSA, the inflation required for nonresponse, and the likely possibility of implementing a 
two-phase sampling approach (described below), it is expected that the opportunities for overlap 
control will be greatest in the national panel, and that the differences in the relevant recruitment 
costs will be the most significant factor in evaluating the benefits of this technique. 


Two-Phase Sampling 

A two-phase sampling approach could be used to build a better sampling frame for DAWN. In 
the first phase, a larger-than-needed sample of hospitals would be drawn, and counts of DAWN 
episodes could be obtained for a few months within each of those hospitals. The second phase 
would then use this information to derive new and more accurate measures of size, which would 
then be used to draw a sample of units that would be most beneficial to DAWN. In other words, 
two-phase sampling allows some opportunity to assess the likely yield of any given unit before 
enlisting it into the system, and the sample design benefits from information that is not otherwise 
available. This approach would be more costly in the short term but could also prove cost effective 
in the long term. 

The redesign team researched the possibility of two-phase sampling using the current DAWN 
data. The feasibility of two-phase sampling depends on the following: 

■ Predictive ability of the 1 st phase data 

■ Relative 1st and 2nd phase data collection costs 

An attempt was made to measure the predictive ability of a few months of data vis-a-vis 
annual reported counts of events. The results were encouraging. Depending upon the estimate of 
interest, simple regression models yielded r-square measures of 0.80 to 0.98 (see Table 3-14), 
indicating that a large amount of variance in the annual measure was explained by the few months 
of data. 
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The regression model used was as follows: 


Y ijk~ B Ojl +B \jl X ijl 


Where 


Yy h = The total annual estimated count of episodes for unit /', estimate j, 12 month 

period k, 

Bqji = The intercept term for estimate j, 2 month period /, 

B X ji = The coefficient term for estimate j, 2 month period /, 

Xyi = The reported count of episodes for unit /, estimate j, 2 month period /. 


Table 3-14. Minimum, Mean and Maximum R-square Measure for Two-Phase Models 


Estimate 

Min. 

Mean 

Max. 

ED Visits 

0.9684 

0.9792 

0.9862 

Total episodes 

0.9601 

0.9696 

0.9768 

Cocaine episodes 

0.9622 

0.9710 

0.9860 

Heroin episodes 

0.9337 

0.9459 

0.9679 

Marijuana episodes 

0.9263 

0.9544 

0.9817 

Methamphetamine 

episodes 

0.7258 

0.8343 

0.9341 


The relative 1st and 2nd phase data collection costs will determine if the 1st phase’s predictive 
ability can be utilized in a cost-effective way. This will depend on the cost structure of the new 
implementation and should be evaluated at that time. 


Sampling ED Visits within Hospitals 

The current DAWN sample design uses a stratified, single-stage cluster sample of hospitals 
and reviews all ED visits for DAWN-reportable episodes within selected hospitals. It was 
recommended that a two-stage design be evaluated in a cost vs. variance framework as an 
alternative to the single-stage design. Two-stage designs would involve the sampling of records 
within selected EDs (e.g., sampling records each day or taking all records from within selected 
days). 

A proper two-stage sampling evaluation would be very similar to the two-phase evaluation 
described above and would yield similar results. It is expected that the prospects for the feasibility 
of two-stage sampling would be good for the more common estimates of interest. However, it 
would be expected that the prospects for sampling the increasingly rare estimates of interest would 
not be good at all. Since DAWN provides estimates for a large number of substances, (including 
not only those that have specific precision constraints for the sample redesign but also those 
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considerably less frequent), the redesign team recommend against two-stage sampling, as it would 
be to the detriment of identifying new and emerging drugs of abuse. 


Merger/Demerger Protocol 

A clear data collection protocol and data adjustment method that acknowledge the possibility 
of mergers and demergers and are informed from a sampling and estimation point of view are 
conspicuously absent from DAWN. Although too late to implement under the current design, such 
a protocol and adjustment method should be incorporated in the new sample design and in the 
new data collection protocol. 


ADJUSTMENTS FOR MISSING DATA (IMPUTATION) 

DAWN benefits from two relatively rare characteristics of sample surveys. First, DAWN 
consists of a panel sample of hospitals with 100 percent overlap between time periods (with the 
exception of sample maintenance selections). Second, monthly ED visits and (to a lesser extent) 
drug-related episodes are relatively stable across a short time period. These two characteristics 
present opportunities for developing procedures for adjustment of unit and item missing data. The 
use of past data for a facility, be it available data for the year under processing or data 
corresponding to the missing months in earlier years, should be explored. 


Nonresponse Adjustment 

The current nonresponse adjustment procedure uses characteristics at the time of sampling to 
form adjustment cells and does not take the size (e.g., ED visits) of nonresponding units into 
account. Changing both of these aspects could improve the bias correction capability of the 
nonresponse adjustments. The specific nature of these changes will depend, to some extent, on 
the variance estimation strategy and sample design actually implemented. Note that changes in 
unit characteristics between the time of sampling and estimation are customarily taken into account 
at the estimation stage, via nonresponse adjustment or post-stratification that considers the current 
characteristics. 


Ratio Adjustment 

The current estimation strategy uses a “benchmark” or ratio adjustment method to bring 
DAWN estimates of ED visits in line with known control totals derived from AHA data. This 
adjustment is done at the DAWN area level. A “no weights less than 1.0” rule is used, which tends 
to dampen the effect this adjustment has, but only when the DAWN estimate exceeds the AHA 
total. Weights can be less than 1.0 due to units selected with probabilities close to or equal to 1.0, 
in strata with little nonresponse adjustment, and in areas where the weighted DAWN estimate of 
ED visits after nonresponse adjustment is greater than the corresponding AHA control total. The 
redesign team recommended that SAMHSA lift this restriction, as that is consistent with the original 
decision to use the AHA data for ratio adjustment. 
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Variance Estimation 


The current variance estimation scheme uses a Taylor series approximation appropriate for a 
stratified, single-stage cluster sample where the clusters are sampled with equal probability within 
stratum. This approach is appropriate for the proposed new design. 


Sample Maintenance and Panel Rotation 

The DAWN sample must allow units new to the population to have a chance of selection. This 
has been traditionally accomplished through the annual sample maintenance process. A panel 
rotation scheme could be used to keep the sample fresher in a more comprehensive way, allowing 
units that have become ineligible to be removed and replaced by eligible selections as well as 
allowing units new to the population to have a chance of selection. However, a panel rotation 
scheme would have periodic recruitment cost implications that should be considered. 


Phasing In Strategies 

SAMHSA requested that the redesign team develop options and make recommendations for 
“phasing in” the new sample. This request was motivated by the concern that SAMHSA would not 
be able to afford to field the new DAWN sample all at once, especially if the total sample sizes 
required were near those being projected for the most ambitious scenarios. The proposed new 
design (with facilities in 48 MSAs, 27 of which will be new to DAWN), can benefit from a phasing-in 
strategy. It is important that these options be considered up-front in the new data collection and 
analysis implementation, as these options have implications for other activities including overlap 
control and the panel rotation/sample maintenance scheme. 

SAMHSA’s objectives for “phasing in” a new DAWN sample were stated as follows: 

■ Control the start-up costs associated with fielding a new sample. 

■ Minimize the discontinuities created by “phasing in” the new sample. 

■ Reach analytic objectives as soon as possible. 

Although not well documented in the statistical literature, the current practice of “phasing in” a 
new sample appears to be as follows: 

■ New samples are often phased in stratum-by-stratum. 

■ The “phase in” period is usually kept as short as possible. 

■ New samples are often run in parallel with old samples for a short period of time to 
measure discontinuities. 

Because DAWN’S primary stratification is at the MSA level, the practice of phasing by strata 
can be followed by definition. The second practice is driven by concerns about how quickly a 
design can become inefficient and is relevant here. Given the discontinuities that will be inherent in 
the implementation of the new DAWN design (new DAWN area definitions, new samples within 
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existing DAWN areas, new case definition, etc.), there is little apparent reason to consider the third 
practice given above. 

There would appear to be two general options for “phasing in” the new sample. The first option 
would phase in by sample size across all targeted MSAs. The second option would phase in by 
MSA. These options and their advantages and disadvantages are described in more detail below. 

“Phasing In” by Sample Size. Recruitment of facilities could occur in all targeted MSAs and 
the national panel in year 1 of the new design, but at initially reduced sample sizes in order to 
control start-up costs. The sample size required for full precision would be obtained after a number 
of years with periodic increases in sample size. The advantage to this approach is that all targeted 
MSAs would be included from the start. However, the primary disadvantage is that full precision 
for each targeted MSA and the national panel would not be achieved until the final year of the 
phase-in period. 

“Phasing In” by MSA. Alternately, DAWN could begin recruitment of the full complement of 
sampled units in only some of the targeted MSAs and the national panel in the first year of the new 
design. Each year, additional MSAs would be brought into the system, also at the level of 
participation needed to achieve the desired precision levels. The advantage to this approach is 
that reasonably precise estimates are available for each MSA in their initial year of participation in 
DAWN. The disadvantages include the need to delay data collection in some MSAs until later 
years; the creation of a fluid stratification scheme, which has variance implications for the national 
estimate; and, in the worst case, the creation of a permanent discontinuity at the MSA level if the 
full design is never realized (i.e., in the face of unexpected funding or other constraints). 

Having considered these options, the redesign team recommended that SAMHSA seriously 
consider the advantages and disadvantages of each approach prior to implementing the new 
design, considering how these options and limitations fit with the agency’s overall plans for DAWN. 
It was also recommended that SAMHSA consult with and require collaboration between the data 
collection and analytic components on this issue, as the objectives, constraints, and cost 
implications from one operation could conflict with the other. On balance, it is likely that an 
approach that phases in the new sample by MSA may be best suited to the objectives of DAWN. 


SUMMARY: THE PROPOSED NEW DESIGN 

The purpose of this research was to develop an efficient sample design given SAMHSA’s 
analytic objectives for DAWN and the available resources as they are currently understood and 
anticipated, respectively. The proposed new design is a stratified, single-stage cluster sample 
designed to yield 950 responding units in 48 MSAs and a national panel. This sample size must 
still be inflated for nonresponse. The stratification is by MSA, ownership (public or private), and 
size (up to 4 size strata within each MSA and ownership category, as described above). Units will 
be selected with equal probability within stratum. The design is based on the following 
requirements: 

■ Level - The design will produce estimates at the national and MSA level. 

■ Estimates - The design requires specified precision levels for total episodes, cocaine, 
heroin, and marijuana mentions. 
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■ Precision - The minimum RSEs are 15 percent for national estimates and 10 percent for 
MSA estimates. 

■ Costs - Using the current reimbursement formula, The new design has an expected 
annual direct reimbursement cost of $3.36 million, compared to the current direct 
reimbursement cost of $1.3 million in 21 MSAs. 

The proposed new design satisfies the above requirements through the following features: 

■ Requires an estimated 950 responding sample units; 

■ Includes 48 targeted MSAs; 

■ Includes Alaska and Hawaii in the national estimate; and 

■ Utilizes stratification by MSA, ownership, and size (ED visits). 
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4. REDESIGNING DAWN’S CASE DEFINITION, 
DATA ELEMENTS, AND CASE SCREENING 
PROCEDURES 


G iven the Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) 
legislatively mandated data collection requirements, the analytic interests of the Drug 
Abuse Warning Network’s (DAWN’s) users, and the operational constraints inherent in 
collecting data in emergency departments (EDs), it was essential to assess whether DAWN was 
“asking the right questions” and collecting the most useful and appropriate data. The redesign 
team was asked to conduct a series of limited field studies that would allow the Office of Applied 
Studies (OAS) staff to understand and quantify the types of cases currently treated in EDs, the 
portion of those cases captured by DAWN, the consistency with which different types of cases 
were likely reported to DAWN, and the overall utility of the current case identification 
procedures. Based on the findings of these studies, recommendations were made for changes 
to the DAWN case definition (the rules used to determine a case’s reportability to DAWN), the 
variables collected to describe each case, and the methods used to identify potentially 
reportable cases from among all ED patients. 

This chapter reviews the limitations of the current case definition, the efforts to develop and 
test a proposed new case definition, proposed changes to the data elements (variables) 
collected about DAWN cases, and the likely impact on DAWN of implementing the proposed 
definition. In addition, this chapter reviews various methods currently used to identify DAWN 
cases in participating EDs, proposed changes that would standardize case identification 
methods, and the likely impact of implementing such changes. 


THE CURRENT CASE DEFINITION 

Since DAWN’s inception, reporters in each participating facility have been asked to review 
the charts of patients treated in the ED to identify cases in which the presenting problem was 
induced by, or related to, substance abuse. For numerous logistical reasons, DAWN must 
continue to rely on retrospective chart review to obtain data. Conducting interviews with 
patients awaiting emergency treatment is logistically infeasible and raises numerous 
confidentiality concerns; asking physicians to collect data during the treatment encounter is 
likewise operationally difficult. Finally, without consistent standards for coding the contents of 
medical records, electronic intercept of DAWN data through fully computerized approaches is 
also infeasible at the present time. As a result of these constraining factors, the redesign team 
focused on developing better methods for more reliably identifying DAWN-reportable cases 
using reporters to review medical records in the participating EDs. 

Reporters select from among all ED visits only those in which they determine that the ED 
patient intended to abuse the substance(s) involved, and the abuse of the drug is related to the 
presenting problem. These inclusion criteria are problematic for several reasons: 

■ “Intent to abuse” is difficult to determine because intent is rarely documented in the 
record except for suicide attempts. Emergency care personnel are concerned with 
treating the acute medical conditions of patients coming to the ED; identifying and 
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documenting the various factors contributing to the presenting condition are less 
important in the ED itself. Therefore, complete and accurate documentation of drug 
abuse is unlikely when patients present for conditions other than overdose or 
intoxication. 

■ For reasons related to possible denial of insurance reimbursement, physicians may be 
reluctant to document a relationship between drug use and the presenting problem. 

■ Protocols for the use of toxicology tests vary widely across EDs. In some EDs, 
toxicology tests are ordered for every patient, or for any of a number of specific 
presenting conditions. Other EDs order tests only as needed. In some EDs, test 
results are received while the patient is still in the ED; in other EDs, results are 
returned days later. In addition, except for a few specific substances (e.g., cocaine), a 
positive toxicology test does not necessarily indicate drug abuse, nor does it specify 
the relationship of the substance to the presenting complaint. DAWN reporters have 
inconsistent access to toxicology results and appear to make inconsistent use of them 
in determining the reportability of DAWN cases. 

■ Frequently, reporters can identify a case as intentional but cannot classify it with 
certainty into one of the categories currently used (i.e., it is difficult for a nonspecialist, 
working only from an ED chart, to differentiate “recreational use” from “dependence”). 

Given the reliance on ED chart documentation, the variability in documentation (including 
toxicology screens) across facilities, and the complexities of the current case definition, 
reporters must often make inferences from limited information to determine reportability to 
DAWN. When hundreds of individual reporters are asked to operationalize a case definition that 
relies on inconsistent documentation, inconsistency in applying the case definition is virtually 
assured. In short, the “intent” and “relatedness” criteria make case identification unreliable and 
case classification difficult. A new case definition is needed that will reduce reliance on 
reporters’ judgement, improve the consistency in the cases reported to DAWN, and facilitate 
interpretation of the resulting data. 

In response to this need, the redesign team considered three alternative case definitions: 
the current case definition (intentional abuse of a reportable substance), a narrow “toxic effects” 
definition, and a more inclusive “broad net” definition. The “toxic effects” and “broad net” 
definitions were proposed as alternatives for addressing variability in documentation practices 
across hospitals that make it difficult for reporters to identify cases reliably. 

The “toxic effects” definition proposed to focus only on acute drug-related morbidity, in 
which the reason for the ED visit was explicitly attributed to the toxic effects of drug use, misuse, 
or abuse as documented in the patient’s chart. By contrast, the “broad net” case definition was 
one in which nearly all cases with drug involvement would be reportable. This option was 
inspired by the Consumer Product Safety Commission’s approach to the National Electronic 
Injury Surveillance System - an approach in which reporters are told to report all cases in which 
a consumer product is mentioned and analysts at a central data processing office then review 
and further code the cases as needed. For DAWN, it was hoped that a case definition could be 
developed that would minimize reporter judgment while allowing (but not requiring) analyst 
review of cases after the fact. During late summer and early fall of 2000, data were collected 
from several hospitals to test the viability of these case definition options and to assess their 
advantages and disadvantages relative to the current approach. 
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User Needs 


In addition to assessing the feasibility of implementing an alternative case definition, the 
redesign team also considered the degree to which resulting data would serve the needs of 
DAWN’S key audiences. As described in Chapter 1 of this report, the redesign process began 
with a detailed constituent analysis to assess current interest in DAWN. The redesign team 
identified a diverse set of audiences for DAWN, with a variety of needs and interests in data on 
drug-related morbidity. DAWN’s constituents desire data at both national and community 
(metropolitan area or smaller) levels, and they desire estimates for different types of patient 
populations and conditions, including but not limited to the following: 

■ Licit and illicit drugs; 

■ Drug abuse, adverse reactions, medication misuse, accidental poisonings, malicious 
poisonings, and suicide attempts; and 

■ Patients seeking care for withdrawal, accidents/injuries, chronic effects of drug use, 
and other medical complications related to drug use. 

DAWN’s constituents require data with which to do the following: 

■ Track trends in drug problems across communities, nationally, and over time; 

■ Identify emerging drugs and new drug combinations; 

■ Learn the specific health consequences associated with different drugs; 

■ Make decisions about drug scheduling, drug labeling, and potential for abuse and 
diversion; and 

■ Learn the characteristics of patients seeking medical attention for different drug-related 
conditions. 

Attention to meeting the needs of key users of the DAWN data was central to the redesign 
process. While it was true that DAWN cannot be all things to all people, the project proceeded 
on the belief that many audiences would use DAWN - and would find it more useful - if the 
case definition were adequately redesigned to address some or all of the above concerns. The 
next section reviews the proposed changes to the DAWN case definition that are intended to 
better serve these needs. That section is followed by a more detailed discussion of a field study 
through which the proposed changes were developed. 


PROPOSED REVISION TO THE CASE DEFINITION 

The findings of the project’s field test indicated the following: 

■ The critical problem with the current case definition is the need to determine - 

retrospectively - a patient’s intention when using a drug. ED charts rarely document 
intent. Even if one assumes that any illicit drug use is with intent to abuse, additional 
rules are required to determine whether the drug use was sufficiently “related to” the 
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presenting complaint to warrant reporting to DAWN. Rules for prescription and over- 
the-counter drugs were more difficult to develop, even for a small field test with only a 
few coders. 

■ Although the “toxic effects” definition had theoretical appeal, it did not identify a 
clinically distinct subset of cases. That is, relying on explicit documentation of drug 
toxicity as a contributing factor in the patient’s condition required coders to exclude 
cases that were arguably drug-related. There were many cases without explicit 
attribution to drug abuse that were, in all other respects, identical to cases in which the 
chart noted that the presenting complaint was “secondary to” drug abuse. Emergency 
care staff on the redesign team’s expert advisory panel were unable to agree on what 
constituted sufficient documentation. Further, it appeared that reporters with more 
medical training might be more likely to “know” that a case was drug-related compared 
to reporters with less medical training. This definition did not appear to identify a 
consistent and meaningful subset of drug-related ED cases in a manner that could be 
consistently applied by reporters of varying training, using charts developed by staff 
with varying documentation styles, across DAWN’S many participating facilities. 

■ The “broad net” definition captured nearly all of the cases included in the current case 
definition, all of the cases included in the “toxic effects” definition, and a number of 
additional drug-related cases. Although a few exclusion rules were needed to set the 
boundary of the definition, these rules were relatively easy to develop and implement 
consistently. Because the “broad net” definition captures a wide variety of cases, new 
data elements (variables) are needed to sort and describe the cases reported. 

Given the needs of DAWN’S diverse constituency, the background of current and likely 
DAWN reporters, information available in ED charts, and the desire to develop simple reporting 
rules that include a clinically distinct set of cases, it was recommended that DAWN’S new case 
definition “cast a broad net.” This broad net should include (with a few specific exceptions) 
every case in which drugs, alcohol, or non-pharmaceuticals are part of the problem being 
addressed. In other words, sections of the medical record describing the presenting problem, 
assessment and/or diagnosis would refer to drug ingestion. New data elements would supply 
the means to subset the “catch” into categories of particular interest to different audiences. 

Rule: Under this proposed case definition, a case would be reportable to DAWN if the 
presenting complaint, clinical assessment, 1 or diagnosis indicates that the patient has used 
drugs, alcohol, or other substances. Excluded from consideration are drugs recorded in a list of 
current medications, since this list includes substances that are not necessarily problematic for 
the clinical management of the patient. Notably, the proposed case definition eliminates both 
“intent to abuse” and “relatedness” as criteria for reportability. If the presenting complaint, 
assessment, or diagnosis sections of an ED chart have documentation of the use of a 
substance - whether or not the patient intended to abuse it - then the case would be reportable 
to DAWN. This change opens the door for inclusion of adverse drug reactions, malicious 
poisonings, and accidental poisonings that are not currently collected in DAWN. 


1 Included in the assessment would be any positive toxicology screen indicating the presence of a drug, subject to the exceptions 
noted. 
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Exceptions: There are a few exceptions to this general rule. Under the proposed case 
definition, an ED visit is not reportable to DAWN if the following occurs: 

■ There is no evidence of current drug use (e.g., Drug use is documented only as 
“history of” or is documented in the social history only); 

■ Alcohol is the sole substance documented and the age of the patient is 21 or older (i.e., 
Underage drinking will be reportable); 

■ A non-pharmaceutical is the sole substance and it has not been inhaled; or 

■ The visit is related to adjustment of a patient’s medication levels. 

These exceptions were developed for the following reasons. First, DAWN is designed to 
track trends in current drug abuse, not in the long-term effects of drug use (which could be 
moderated by any number of unrelated factors). For example, differences in documentation 
styles would impact the identification of chronic illnesses caused by past drug use. Since ED 
physicians are concerned with treating the presenting problem, they do not always document 
the underlying cause of the illness, particularly if the cause carries with it social stigma. 

Perhaps the best example of this is HIV/AIDS, which has many root causes, of which IV drug 
use is only one. Poor documentation of drug involvement in these cases, coupled with inter¬ 
reporter differences in making assumptions about drug involvement, lead to the 
recommendation to exclude such cases unless current drug use is noted in the record. The 
tracking of chronic illnesses and their etiology is better accomplished by the Centers for Disease 
Control and Prevention’s (CDC’s) surveillance efforts and should not be duplicated in DAWN. 

Second, this case definition recommendation attempts to address some of the requests of 
DAWN’S audiences for data on cases involving only alcohol. As reported later in this chapter, 
analyses of data from a field study and from the National Electronic Injury Surveillance System 
(NEISS) suggest that adding “alcohol only” cases to DAWN would overwhelm the system, as 
much as tripling the number of cases reported. Additionally, findings from the study of the 
impacts of health system change (see Chapter 2) show that a large number of “alcohol only” 
patients are seen in the ED as a result of local policies to manage the problem of public 
intoxication. Variation in these policies across cities and over time are likely to have a notable 
influence on observed trends. Thus, trends in adult alcohol cases in DAWN are more likely to 
reflect changes in local policies than in underlying levels of morbidity. 

Plowever, DAWN is the only major substance abuse data collection effort that does not 
include alcohol among the list of drugs surveyed. There is a definite need to monitor the effects 
of alcohol abuse, particularly among youth (under age 21), for whom alcohol is an illegal 
substance. Visits to the ED by underage drinkers are more reflective of morbidity than are visits 
by older drinkers, many of whom are “chronic inebriates” seeking temporary shelter. The data 
also suggest that the volume of underage drinking cases will be manageable in terms of 
reporter burden. Therefore, as a compromise between user needs and resource constraints, it 
was recommended that underage drinking be added to the set of cases collected by DAWN, but 
that “alcohol only” episodes for patients of legal drinking age should continue to be excluded. 

Third, analysis of data from NEISS suggests that expanding the case definition to include 
“any use of a substance” potentially introduces a large volume of cases involving exposure to 
household products and hazardous materials. For the most part, collection of data on the health 
consequences associated with non-pharmaceuticals is successfully accomplished by NEISS, 
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and that effort should not be duplicated in DAWN. However, substances of abuse emerge 
unexpectedly, and chemicals and household products are sometimes abused, typically by 
inhalation. Therefore, it was recommended that the use of non-pharmaceuticals be reportable 
to DAWN if they were inhaled. Non-inhaled, non-pharmaceuticals (e.g., burns caused by spilled 
chemicals or cleaning supplies ingested by toddlers) would not be included in the proposed 
case definition. 

Finally, in the process of reviewing data in the field test, it was determined that a rule must 
be developed to exclude ED visits related to medication adjustment. DAWN is designed to 
monitor adverse health consequences associated with drugs. Because of metabolic differences 
among patients, some may experience adverse health consequences during a period in which 
their physicians are attempting to stabilize them on a new medication. Sometimes these 
symptoms result in ED visits. Because these reactions are, in effect, part of the normal course 
of the therapeutic process, they fall outside the intended scope of DAWN. The most prevalent 
example of this kind of case involves hypoglycemic episodes due to insulin reactions in 
diabetics. Because the field test was implemented in only a few hospitals and only a handful of 
these cases were encountered, better specification of this exclusion rule may not be possible 
until the new definition is actually implemented. 


Proposed Changes to the Data Processing Approach 

Recommendations from the field test included two important changes to the way DAWN 
data are processed. First, collection of a brief “case description” for all reported cases was 
proposed. This would be an open-text field in which the reporter would be asked to record the 
presenting problem and assessment(s) as noted in the ED chart. Such information is not 
currently available in DAWN, and central data processing staff has no indication from the 
current report forms whether a submitted case actually met the reportability criteria. The 
proposed case description field would permit data processing staff to double-check the 
reportability of cases, to identify common circumstances in which reporter errors occur, and to 
identify circumstances associated with emerging drug problems. 

A second proposed change is to begin thinking about cases in terms of several distinct 
types. Since its inception, DAWN has reported all cases in the aggregate as “drug abuse” 
cases, which was appropriate given the case definition in use. With the proposed broad net 
case definition, DAWN will continue to collect primarily drug abuse cases, but it will also include 
cases such as adverse drug reactions in which no abuse was intended. For proper analysis 
and interpretation, DAWN’S varied audiences will require cases to be disaggregated into the key 
types included in the proposed case definition. Thus, it was recommend that reporters classify 
reportable cases into types using the flow chart below. 

Based on the field study data and the interests of DAWN’s users, there appear to be eight 
different “types” of reportable cases: 

■ Suicide attempts - defined strictly as documentation of “suicide attempt” or “attempted 
to kill self” by means of a drug overdose. Suicidal gestures, suicidal ideation, or 
ingestion of multiple pills are not sufficient to be included in this category. 

■ Seeking detox - a category made necessary by hospital administrative policies that 
may require patients to obtain medical clearance in the ED for substance abuse 
treatment. These cases are characterized by documentation that the patient is seeking 
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“detox,” “rehab,” “medical clearance” or “help for a drug problem.” This category, 
coupled with the “disposition from ED” variable, will allow OAS to track the effects of 
ED administrative policies on DAWN estimates over time. 

■ Underage drinking - defined as documentation of alcohol use in the absence of any 
other substances in patients under 21 years of age. 

■ Adverse drug reaction - defined as any case with documentation of a reaction to a 
prescription or over-the-counter drug and no evidence that the drug was used counter 
to direction. These cases include allergic reactions to drugs, drug/drug interactions, 
and instructions to discontinue use of a prescription or over-the-counter drug. 

■ Overmedication - to include cases with documentation that patients took more of a 
prescribed or over-the-counter drug than instructed, either because they forgot they 
had already taken a dose, they were trying to make up for a missed dose, or because 
symptoms did not subside with the recommended dose. 

■ Malicious poisoning - to include cases in which a patient is poisoned by someone else; 
this category includes the “drug rape” cases many users assume to be included in 
DAWN. 

■ Accidental poisoning - to include cases in which children ingest pills or other 
substances by mistake, or when individuals ingest a drug not realizing its nature. 

■ All other drug abuse - intended as a “catch all” category to include all other cases not 
classified above. These cases can be further categorized using the data elements 
(variables) proposed for collection. 

The “type of case” classification (see Figure 4-1) is important for subsetting different 
categories of cases that will be reported under the new DAWN. These can be defined to be 
mutually exclusive but prioritized categories, and the decision boxes in the following chart 
provide short-hand criteria for each. It was recommended that each of the first seven categories 
be defined very narrowly; it is expected that there would be relatively few cases in any one of 
these categories. All other cases (particularly the large number for which determining intent is 
impossible) would filter down into a general drug abuse category. 


DESCRIBING CASES IN THE “BROAD NET” 

Currently, DAWN collects a limited number of data elements (variables) with which to 
describe cases. Among the limitations of the current approach, DAWN collects no information 
on the medical conditions for which drug users seek treatment in EDs. Given the proposed 
expansion of the case definition to include a broader array of cases, it will be important to 
include data elements that will allow DAWN’S users to subset different groups of cases that may 
be of particular analytic interest (e.g., to separate adverse reactions to prescription drugs from 
illicit drug overdoses). To accomplish these objectives, the redesign team recommended a 
series of changes to the data elements collected on DAWN cases. 
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Figure 4-1. Type of Case Classification. What type of case is it? Classify the case as the 
first type for which it meets the specified criteria. Make your decision based on documentation 
in the chart. Do not assume or infer. 
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The proposed data elements fall into four general categories: 

■ Operations Data - these elements include items necessary for processing DAWN 
cases, including the provider number, reporters’ cross-reference numbers, and 
date/time of ED visit. 

■ Demographic Data - these include basic patient information such as age, sex, 
race/ethnicity, and living arrangements. 

■ Drug Data - several fields will be available to record the drugs implicated in the ED 
visit. Reporters will be asked to record only those substances with documented 
relevance to the ED visit, not all of the substances “on board” or for which a patient has 
a current prescription. 

■ Case Description Data - these data elements represent the most significant change to 
the DAWN data collection process, as they will provide substantial detail on the 
medical conditions for which patients seek treatment, the type of case, and disposition 
from the ED. 

Table 4-1 describes each of the proposed changes to the DAWN data elements, including 
items to be added, changed, deleted, or retained from the DAWN data collection form. A brief 
description of the reason for the proposed changes is provided for each element. 


APPLICATION OF THE PROPOSED CASE DEFINITION AND DATA ELEMENTS 

This section details findings obtained in the redesign team’s field test of the proposed case 
definition and data elements. The field study was structured to assess the feasibility of different 
case definition options, to develop and clarify coding decisions, and to estimate the likely 
change in the volume of cases with the adoption of the new, proposed case definition. 


Procedure 

In order to test the feasibility of the broad net case definition and data elements, members 
of the redesign team visited five hospital EDs currently reporting data to DAWN. Three of the 
DAWN hospitals were in inner city settings and were known to treat a high number of substance 
abuse cases. One suburban hospital was expected to provide access to fewer substance 
abuse cases but more alcohol and adverse drug reactions. The other suburban hospital was 
known to treat a high number of substance abuse cases due to the fact that psychiatric cases 
from the surrounding area were sent preferentially to its psychiatric emergency room. As part of 
a special study on case identification methods (reported later in this chapter), researchers at a 
large university hospital were asked to collect information about substance-related visits treated 
in their ED. “University Hospital” is an inner city hospital known to treat a large number of cases 
involving substance abuse. Thus, these six hospitals were a convenience sample, selected to 
maximize the number of cases involving different kinds of substance abuse. 
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Table 4-1. Recommended Data Elements for DAWN (ED Component) 


Data Element 

Recommended change 

Explanation 

Provider number 


A unique hospital identifier, for SAMHSA use only. 

Cross-reference 


A record identifier for reporter use only. 

Date and Time of ED Visit 

Month of visit 

For SAMHSA use only. Date/time of visit, when combined with other 
case characteristics, could potentially identify an individual. Month of 
visit is the smallest unit needed to process and report DAWN data. 

Age 

Remove age limits 

DAWN should collect data for patients of all ages. To protect 
individual identities, age categories should be collapsed for all 
patients under age 6 and over age 85 when reported. 

Sex 

None 

To be retained. 

Ethnicity/Race 

White/Black/Hispanic/Other 

Reporters have substantial difficulty with the ethnicity category (not 
understanding it, or the information is not available). Recommend 
reverting to the 4-category combined race/ethnicity variable. 

Patient’s Home Zip Code 

Living arrangements (fixed address, no 
fixed address, not documented) 

Zip code data are often unavailable, or require reporters to access 
separate databases. The data are not available for public use, and 
can potentially be used to identify individual patients. Because 
policies toward the homeless can have a substantial effect on ED 
case volume, the recommendation is to change this variable to focus 
on whether or not patients are homeless. 

Case description 

Open-text field in which reporters record 
information about the presenting complaint 
and assessment. 

This is a new data element. For SAMHSA use only. 

Reason for Taking 
Substances 

Do not collect. 

Eliminate from current form. Replace with “Type of case” (see 
below). 

Type of Case 

Eight categories, described in flow chart 
above. 

This is a new data element. 

Reason for Present Contact 

Do not collect. 

Eliminate from current form. Replace with “Presenting complaint” 

(see below). 

Presenting Complaint or 
Condition 

Categories to identify the type of medical 
condition for which the patient sought 
treatment in the ED. To be developed. 

This is a new data element. Findings from the field study suggest an 
initial list of conditions, but the data collection contractor should 
monitor information in the “case description” variable over the first 
several months of data collection to develop and refine a 
comprehensive and analytically useful list. 

Diagnosis(es) 

Text field; record all listed diagnoses. 

This is a new data element. Diagnoses could be categorized as 
needed to meet specific research questions. 

Alcohol involved? 

None 

To be retained. 
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Table 4-1. Recommended Data Elements for DAWN (ED Component) (continued) 


Data Element 

Recommended change 

Explanation 

Drug(s) involved - list all 
substances separately 

Allow reporters to list up to 6 substances 
plus alcohol (currently 4 + alcohol) 

Addition of Rx/OTC reactions may necessitate these additional fields. 

Form in which drug was 
acquired 

Do not collect. 


Route of administration 

Change to include only four categories: 
injected, oral, 

inhaled/sniffed/snorted/smoked, and 
information not documented. 

For SAMHSA use only. This data element will be used only to 
facilitate correct classification of nonpharmaceuticals (i.e., inhalants), 
as well as to differentiate prescription and over-the-counter medicines 
available in different forms (e.g., antihistamine pills versus nasal 
sprays). 

Source of substance 

Do not collect. 


Disposition 

Revise to include more specific categories: 

(a) Treated and released (discharged 
home, transferred, sent to drug 
treatment/detox, released to police 
custody); 

(b) Admitted (ICU/critical care, 
medical/surgery, psychiatric unit, 
chemical dependency/detox unit); 

(c) Left against medical advice; 

(d) Died; and 

(e) Information not documented 

In recent years, about 48% of DAWN cases were “treated and 
released,” while another 48% were “admitted.” More information is 
needed to differentiate patients in these categories. In particular, 
more information is needed to indicate when the ED is a route of 
entry into the substance abuse treatment system. 

Coded remarks 

Do not collect. 


Urgency / Acuity 

Do not collect. 

The field test indicates that because of substantial variation in ED 
triage and documentation procedures, these data cannot be collected 
using a standard set of categories. 

Insurance coverage 

Do not collect. 

The field test indicates that this information is not readily available to 
DAWN reporters. 

Source of information 

Do not collect. 

The field test indicates that because of substantial variation in ED 
treatment and documentation practices, it would be operationally 
difficult and analytically meaningless to ask reporters to document 
what sections of the chart contained information about the 
circumstances of the case. The “case description” variable should 
provide sufficient detail for analytic and quality control purposes. 



Research staff visiting each of the hospitals reviewed the entire ED chart for each patient 
treated in the ED during a specified period of time. The period of time varied among hospitals 
due to differences in patient volume; charts were reviewed for as many days as feasible during a 
2- or 3-day site visit. Because the field test also needed to assess the appropriate boundaries 
for the “broad net” definition, data were collected for every case with any mention of a substance 
in the ED chart, including consumer products such as bleach. By being overly inclusive at this 
stage, the field test allowed the research team to test different definitions and decision rules on 
the same set of data. In order to better understand what kinds of cases were being treated in the 
ED, several data elements were collected for each substance-related case, including a verbatim 
case description based on the presenting complaint, history of presenting illness, clinician 
assessment, diagnosis, and disposition. 

The research team then coded the case description fields for a variety of characteristics that 
were gradually refined to become the proposed case definition and data elements. Coders were 
trained on the rules for the current case definition, the proposed case description, and the 
proposed data elements. Each case was double-coded by two coders working independently, 
and differences were reconciled by discussion among the research team and with OAS. These 
discussions helped to clarify issues associated with coding and further refined coding rules for 
the data elements. Throughout the field study, a decision on including alcohol-only cases was 
under consideration. As a result, data for both adult and underage drinking are included in the 
data shown here. 


Results 

Of 8,157 records reviewed at the six hospitals, 277 met the current case definition criteria, 
and 333 met criteria for the proposed broad net case definition. A total of 248 cases involved 
documentation of current alcohol use but no other current drug use. There is some overlap 
between the “Alcohol Only” cases and the other DAWN cases: three alcohol-only cases met 
criteria for the current case definition because they involved documentation of past drug use. 
Only 20 alcohol-only cases involved underage drinking; in the following tables, these 20 cases 
are included in both the alcohol-only columns and the proposed case definition columns. 

Table 4-2 displays the 10 most common diagnoses detected under the current case 
definition, the proposed case definition, and in cases involving the use of alcohol alone. As 
shown, the major difference in diagnosis between the current case definition and the proposed 
case definition is the adverse drug reactions included in the proposed new definition. Notably, 
among alcohol only cases, a very high proportion (60%) carry a diagnosis of intoxication, 
whereas very few cases in the current and proposed case definition carry this diagnosis. 

Table 4-3 displays several categories of cases and shows whether they are included in the 
current and proposed case definitions. The purpose of this table is to illustrate some of the 
information used in establishing rules for reportability, as well as in developing rules for the 
proposed “type of case” data element. Of note, the proposed case definition adds adverse drug 
reactions, overmedication, malicious poisoning, and accidental poisoning. With very few 
exceptions, all cases in these categories are new to DAWN under the broad net definition. 
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Table 4-2. Ten Most Common Diagnoses in Drug-Related Cases (Not mutually exclusive) 


Diagnosis 

Current Case 
Definition 

Proposed Case 
Definition 

Alcohol Only 

N % of Cases 

N % of Cases 

N % of Cases 

Substance Abuse 

75 

27.1% 

79 

23.7% 

37 

14.9% 

Intoxication 

41 

14.8% 

50 

15.0% 

149 

60.1% 

Depression 

36 

13.0% 

36 

10.8% 

5 

2.0% 

Suicidality 

30 

10.8% 

30 

9.0% 

1 

0.4% 

Drug or Alcohol 
Dependence 

23 

8.3% 

23 

6.9% 

6 

2.4% 

Withdrawal 

22 

7.9% 

22 

6.6% 

11 

4.4% 

Overdose 

16 

5.8% 

16 

4.8% 

0 

0.0% 

Cellulitis/Abscesses 

16 

5.8% 

13 

3.9% 

0 

0.0% 

Toxicity 

3 

1.1% 

7 

2.1% 

0 

0.0% 

Adverse Drug 

Reaction 

1 

< 1% 

19 

5.7% 

1 

0.4% 


Table 4-3. Categories Included in Current and Proposed Case Definitions 


Category 

Number of 
Cases 

Included in Current 
Case Definition? 

Included in Proposed 
Case Definition? 

Suicide Attempt by Drug Overdose 

5 

Yes 

Yes 

Seeking Detox 

49 

Yes 

Yes 

Allergic/Adverse Reaction to Drugs 

Taken For Medical Condition 

45 

2 of 45 (ADRs along 
with illicit drug use ) 

Yes 

Overmedication 

5 

3 of 5 (took additional 
meds to enhance 
effects) 

Yes 

Alcohol/Drug Interaction 

0 

No 

Yes 

Alcohol Only 

Under 21 years old 

Over 21 years old 

20 

228 

3 of 228 with a history 
of substance abuse 

Yes 

No 

Malicious Poisoning 

2 

1 in conjunction with 
illicit drug use 

Yes 

Accidental Ingestion 

0 

No 

Yes 

Inhaled Substances (other than drugs) 

If intent to abuse 

Other 

0 

Yes 

No 

Yes 

No 

History of Drug Use 

7 

Yes 

No 

All Other Cases with Substance Use 

207 

Yes 

Yes 

TOTAL 

277 

333 


Table 4-4 shows the breakdown of cases as categorized by the proposed “type of case” data 
element. As expected, there are relatively few cases in most of the first seven categories, as 
these are defined narrowly. The variable appears particularly useful for disaggregating three key 
categories of interest - underage drinking, adverse reactions, and patients who are seeking 
detox. For analytic purposes and in the interests of DAWN’S audiences, the ability to readily 
identify these cases is important. Although there are very small numbers of cases in the 
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remaining categories, it is recommended that they be maintained because some analysts may 
find them useful and because the field study data are not representative of all DAWN hospitals. 
The substantial proportion of cases filtering down into the “other drug abuse” category is not 
surprising given the type of information DAWN collects. Use of other proposed data elements 
can help analysts to further subset these cases (e.g., to identify psychiatric emergencies or 
inhalant abuse). 


Table 4-4. “Type of Case” Results from Field Study 


Category 

Number of Cases 

Percent of 



Total 

Suicide Attempt by Drug Overdose 

5 

2% 

Seeking Detox 

49 

15% 

Underage Drinking 

20 

6% 

Allergic/Adverse Drug Reaction 

45 

14% 

Overmedication 

5 

2% 

Malicious Poisoning 

2 

1% 

Accidental Poisoning 

0 

0% 

All Other Cases Involving Substance Use 

208 

62% 

TOTAL 

333 

100% 


Table 4-5 displays some common conditions associated with the proposed case definition. 
There is a high degree of overlap among these categories, and they are not mutually exclusive. 
Note that there are many cases involving psychiatric emergencies among the data involving 
drugs, and many intoxication cases among the “alcohol only” cases. These findings emphasize 
the importance of psychiatric consequences associated with drug use, and that alcohol-only 
cases are dominated by cases of intoxication, rather than by more serious medical conditions. 

Monitoring the kinds of conditions treated is useful for interpreting trends in DAWN data, 
since some conditions are known to be treated in places other than EDs. The number of cases 
treated in the ED for less urgent conditions depends on a local health care system that is 
vulnerable to change. DAWN currently provides no information about the medical conditions 
associated with drug-related ED visits. A “presenting complaint” variable could therefore be very 
useful in helping to describe the ways in which drug abuse manifests itself in EDs over time. The 
list of conditions in Table 4-5 is a preliminary list based on a data from a few hospitals; a final list 
can be developed by monitoring the case description field for the first several months after the 
new case definition is implemented. 
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Table 4-5. Selected Conditions Associated with Cases 



Current Case 
Definition 

N % 

Proposed Case 
Definition 

N % 

Alcohol Only 

N % 

Toxic Ingestion/ Overdose 

32 

12% 

32 

10% 

0 

0% 

Seizures 

8 

3% 

10 

3% 

9 

4% 

Chest Pain 

20 

7% 

21 

6% 

5 

2% 

Psychiatric Emergencies 

94 

34% 

98 

29% 

10 

4% 

Accidents/Injuries 

38 

14% 

43 

13% 

38 

15% 

Intoxication 

42 

15% 

40 

12% 

166 

67% 

Organ Damage 

1 

0% 

0 

0% 

10 

4% 

Conditions Associated with 
Administration of Drugs 
(e.g., injecting) 

27 

10% 

23 

7% 

0 

0% 

Withdrawal 

31 

11% 

31 

9% 

10 

4% 

Premature Labor 

3 

1% 

3 

1% 

1 

0% 

Seeking Detox 

52 

19% 

52 

16% 

21 

8% 


Table 4-6 displays the disposition of cases captured under the current and proposed case 
definitions, as well as the cases involving alcohol only. Notably, fewer alcohol only cases are 
admitted to inpatient settings from the ED than cases identified under the current and proposed 
case definitions. A full 62 percent of alcohol-only cases were treated and released, indicating 
again that these cases are less medically urgent than cases captured under the current and 
proposed case definitions. 


Table 4-6. Disposition of Cases 


Disposition 

Current Cases 

Proposed cases 

Alcohol Only 

N % 

N % 

N % 

Treated and Released, no 
referral noted 

89 

32.4% 

143 

42.9% 

156 

62.9% 


Treated and Released, referral noted: 


-Referred to Medical Clinic 

1 

0.4% 

3 

1.2% 

5 

2.0% 

-Referred to Psych. ED 

7 

2.5% 

7 

2.1% 

1 

0.4% 

-Referred to Drug/Alcohol 
Treatment 

50 

17.8% 

49 

14.6% 

28 

11.3% 

-Referred to Urgent Care 

5 

1.8% 

5 

1.5% 

3 

1.2% 

-Referred to Outpatient 
Psychiatry 

2 

0.7% 

2 

0.6% 

1 

0.4% 

Released to Police Custody 

2 

0.7% 

2 

0.6% 

0 

0.0% 

Admitted - ICU 

6 

2.1% 

7 

2.1% 

4 

1.6% 

Admitted - Med/Surg 

59 

21.4% 

62 

18.5% 

18 

7.3% 

Admitted - Psych. Inpatient 

22 

7.8% 

22 

6.9% 

2 

0.8% 

Left Against Medical Advice 

11 

3.9% 

11 

3.3% 

19 

7.7% 

Died 

2 

0.7% 

2 

0.6% 

0 

0.0% 

Other 

17 

6.4% 

14 

4.2% 

7 

2.8% 

Unknown 

4 

1.4% 

4 

1.2% 

4 

1.6% 

TOTAL 

277 

100% 

333 

100% 

248 

100% 
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Finally, Table 4-7 displays information about several data elements that could not be 
consistently operationalized across the hospitals providing data for this field test and the source 
or implications of problems with these data elements. 


Table 4-7. Problematic Data Elements 


Data Element 

Documentation 

Problem 

Ethnicity/Race 

87% missing Ethnicity 

40% missing Race 

Ethnicity and race were sometimes mentioned in the clinical 
notes, but data were not consistently documented and it was 
unclear whether the same categories were consistently 
applied. Although race is arguably important for 
understanding drug abuse trends, it is unlikely that ethnicity 
data will be able to be collected with any degree of 
consistency across hospitals. 

Zip code 

49% missing 

The research team often did not have access to billing data 
that contained this data element. 

Homelessness 

Unlike race and zip code, 
it is not possible to tell 
when homelessness is 
true but undocumented. 

As expected, homelessness was not common, but was 
occasionally noted in the chart (for instance, it was noted 
that the patient was released to a shelter). Homelessness 
was sometimes documented in the clinical notes, even 
though there was an address associated with the record. 
Although a “homelessness” value may be a useful element 
under a general “living arrangements” variable, analysts 
should note that homelessness will be under-reported 
because this information is not reliably documented. 

Primary Expected 
Source of 

Payment 

85% missing, remainder 
impossible to code 

This variable was not consistently documented in the 
records reviewed. Some of the data were maintained in 
billing databases to which the research team did not have 
access. In other hospitals, the medical records contained 
information about the expected source of payment. 

However, the data were difficult to interpret, for several 
reasons. When documented, charts often included the 
name of a specific health plan or HMO. Without knowledge 
of the patient’s specific plan or plans available in the city, it 
was impossible to classify named plans correctly. A more 
generic list of categories (i.e., public pay vs. private pay) 
was attempted, but this was not useful because several 
commercial HMOs offer coverage under Medicaid and 
Medicare. Finally, in some charts multiple sources of 
payment were documented. 

Acuity of 

Presenting 

Condition 

32% missing, majority of 
remainder impossible to 
code 

This variable was not consistently documented across the 
studied hospitals. When documented, different systems for 
describing acuity were used and they were not comparable. 


Expected Burden of the Proposed Case Definition 

Because the six hospitals used in the case definition field test are not representative of all 
DAWN hospitals, these data cannot be used alone to estimate the likely impact of the new case 
definition. Instead, data from the Consumer Product Safety Commission’s (CPSC) National 
Electronic Injury Surveillance System (NEISS) were used to provide an initial indication of the 
volume of cases likely to be generated under the proposed case criteria. 

NEISS has recently begun collecting data about a wide range of cases treated in the EDs of 
U.S. hospitals. Once restricted to collection of information about consumer products, NEISS now 
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collects information on two classes of cases treated in the ED: (1) all nonfatal injuries and (2) 
medical conditions with a documented external cause. A nationally representative sample of 66 
NEISS hospitals began reporting these data in July 2000. Included in NEISS cases are acute 
conditions related to the use of licit and illicit drugs, use of alcohol, and toxic reactions resulting 
from contact with household poisons or industrial chemicals. Although the NEISS case definition 
does not include all cases that would be reportable to DAWN, the overlap is sufficient to allow 
some meaningful analyses of NEISS data to inform the DAWN redesign. Importantly, NEISS is 
the only ongoing data collection system outside of DAWN that would allow for preliminary 
analyses of this type on a nationally-representative sample of ED cases. 

CPSC provided raw data on 83,888 NEISS cases collected from July 1 to August 31,2000. 
The data file included information about products involved in the presenting condition, as well as 
a diagnosis and an open text field describing the condition. Of these cases, 1,551 met the 
criteria for the proposed new DAWN case definition. NEISS contained some drug-related cases 
that were excluded for a number of reasons. Specifically, all cases in which screening for drug 
use (drug testing) was requested by an employer or parent, but seemed unrelated to a patient’s 
health complaint, were excluded. Also excluded were a number of cases in which substances 
were not directly related to the presenting condition. For example, injuries caused by falling beer 
kegs were not included unless someone had actually ingested the beer. Also excluded were 
cases in which a patient choked on pills. 

Table 4-8 displays the different categories that comprise the proposed new DAWN definition. 
In NEISS, the number of adverse drug reaction cases is nearly two-thirds that of the substance 
abuse cases. This ratio is considerably higher than the ratio found in the field study hospitals. 
NEISS is more sensitive to adverse reaction cases than to conditions related to substance abuse 
because NEISS captures only conditions with an external cause. DAWN cases that involve 
patients seeking detox or psychiatric emergencies, in which substances play a contributing rather 
than a causal role, may not be consistently identified by NEISS. 

Table 4-8. Characteristics of NEISS Cases Meeting the Proposed Case Definition 



DAWN Cases 

Overdose/Abuse of Drugs (or alcohol 
and drugs in combination) 

656 

Adverse Drug Reactions 

422 

Overmedication 

50 

Accidental poisoning (children) 

181 

Inhalation of nonpharmaceuticals 

53 

Underage drinking 

181 

Malicious poisoning 

8 

TOTAL 

1,551 


Another important consideration was the variation in the number of proposed DAWN cases 
by size of hospital. The field study focused on large, high volume DAWN hospitals, and 
therefore provided little insight into how the adoption of the proposed case definition might 
differentially impact hospitals of varying size. CSPC classifies its hospitals into four categories 
based on size. Thirty-one of the hospitals were classified as “small” hospitals, with annual visits 
under 15,730. Nine of the hospitals were “medium” hospitals, with annual visits between 15,731 
and 25,895; six hospitals were “large” hospitals, with annual visits from 25,896 to 42,498. 
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Finally, there were 15 “very large” hospitals with annual ED visits greater than 42,498. Five 
hospitals specializing in care for children participate in NEISS as well. 

Table 4-9 shows that the number of NEISS cases meeting the proposed DAWN definition 
varied widely, from 0 to 157 cases over the two-month period. Large hospitals provided more 
cases than smaller ones, but the standard deviation was quite large in the large hospitals. 


i auic ‘t'j. numui 

Hospital Type 

st i mcvv uaoco uj 

# OF 

HOSPITALS 

f uiac ui nuapuai 

DAWN Cases 
identified 

RANGE OF 

DAWN VISITS 
PER HOSPITAL 

Mean (S.D.) 

Children’s 

5 

175 

0-79 

35 (28.35) 

Small 

31 

266 

0-29 

8.77 (7.68) 

Medium 

9 

159 

2-48 

17.67 (15.22) 

Large 

6 

272 

2-157 

44.33 (57.72) 

Very Large 

15 

679 

0-155 

45.27 (38.53) 

TOTAL 

66 

1,551 

0-155 

23.5 (30.8) 


Besides looking at the total volume of reportable cases, it is also instructive to look at the 
ratio of new cases to current DAWN cases. Such data provide a better understanding of the total 
burden on reporters, who must review all available charts in order to identify cases. Thus, any 
given number of cases will not fully reflect the total burden involved in identifying those cases. 
Table 4-10 shows the number of ED visits in each hospital stratum, as well as the number of 
cases meeting the current and proposed DAWN case definitions. These data are estimated 
based on incomplete raw data provided by CPSC; notably, two hospitals did not provide 
information about the total number of ED visits in the previous year, so these hospitals are not 
included in this table. Also, the total number of visits for some of the hospitals has apparently 
increased since they were originally sampled, and they have not been reassigned to new strata 
to reflect this change. 

Table 4-10 places the ratio of new cases to current cases at about two to one. That is, the 
analysis of NEISS data would suggest that the proposed new case definition will roughly double 
the number of cases reported to DAWN. Overall, one new case is detected for every 216 
records reviewed, although there is considerable variability among hospitals. These figures are 
likely to be slight underestimates because NEISS is not likely to detect a portion of DAWN cases 
in which substances play a contributing rather than a causal role. 

Analyses of data from six DAWN hospitals and from hospitals participating in NEISS, 
suggested that proposed the new case definition was readily operationalized. These data helped 
clarify boundaries (i.e., exclusion criteria) for the proposed definition, and provided guidance for 
development or exclusion of new data elements. The proposed new case definition improves 
upon aspects of the current case definition. It does not require coders to speculate about the 
intent involved in the ingestion of a substance or the medical relationship between the condition 
being treated and the ingested substance. 


116 


Development of a New Design for DAWN 



Table 4-10. Estimated ED Visits per Reportable Case (NEISS Data, July-August, 2000) 


Stratum 

# 

Sites 

Estimated 

Number of ED Visits 

Est. ED 
Visits in 2 
months 

Proposed Case Definition 

Current Case Definition 

Number 

New 

Cases 

ED Visit 
per 
Case 

Cases 
per ED 
Visit 

Number 

Current 

Cases 

ED 

Visit 

per 

Case 

Cases 
per ED 
Visit 

Total 

Average 

SD 

Children's 

5 

250,910 

50,182 

15,595 

41,818.33 

175 

2,393.0 

0.004 

44 

950 

0.001 

Small 

30* 

270,958 

9,031 

6,237 

45,159.67 

266 

169.8 

0.006 

78 

579 

0.002 

Medium 

8* 

228,186 

28,523 

13,360 

38,031.00 

159 

239.2 

0.004 

64 

594 

0.002 

Large 

6 

393,590 

65,598 

29,195 

65,598.33 

272 

241.2 

0.004 

151 

434 

0.002 

Very 

Large 

15 

866,425 

57,761 

20,488 

144,404.17 

679 

212.7 

0.005 

319 

453 

0.002 

TOTAL 

64 

2,010,069 

31,407 

27,536 

335,011.50 

1551 

216.0 

0.005 

656 

511 

0.002 


* Missing data from one hospital 


While the proposed case definition is likely to substantially increase the number of cases 
reported to DAWN, it also substantially increases the potential applications for those data. The 
recommended data elements will provide more information on the medical conditions for which 
individuals seek emergency care, and will allow analysts some flexibility in subsetting cases of 
interest (e.g., adverse reactions, underage drinking, and child poisonings). 

The proposed new case definition may have implications for how reporters identify cases. In 
order to identify cases effectively and to minimize reporter burden, some hospitals may need to 
employ different screening strategies. The following section describes the two most common 
screening methods currently in use (ED log review and ICD-9-CM screens), and how case 
identification is likely to be affected by use of these methods in conjunction with the proposed 
definition. 


CASE IDENTIFICATION PROCEDURES 

There are three methods now in place for identifying DAWN cases. The standard, preferred 
method is for reporters to review all patient medical records for all ED visits, and then file DAWN 
reports for those meeting the case criteria. (This is often called the “100 percent medical record 
review” method.) The other two approaches involve the screening of medical records to identify 
likely DAWN cases, and then reviewing the full medical record for only those cases flagged by 
the screening method. One method relies on the ED log, which lists patient name, identifying 
characteristics, presenting complaint, and disposition for every visit. Information is entered into 
the log at the initiation of the ED visit. In the “log screen” method, the DAWN reporter examines 
the ED log for cases likely to be related to drugs, then reviews the medical records for only those 
cases. Another method relies on ICD-9-CM codes to screen for potential drug-related cases. In 
that method, the reporter reviews billing records to identify ED visits that were assigned any of a 
number of pre-specified ICD-9-CM codes. Only those medical records associated with specific 
ICD-9-CM criteria are reviewed. 
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Screening methods are typically used in DAWN hospitals that do not to have full access to 
all ED patient medical records, or in hospitals that have a very low ratio of DAWN cases to ED 
visits. For reporters in the latter type of EDs, a screening method helps reduce the substantial 
burden of reviewing large numbers of ED medical records to identify very few cases. While 
screening methods save considerable effort in that fewer medical records need to be reviewed, 
these methods also increase the risk of missing DAWN cases. The performance of these 
screening methods has never been systematically assessed under the current case definition; it 
is not known how well either screening method will perform under the rules of the proposed 
“broad net” definition. The redesign team was asked to conduct a preliminary assessment of 
each screening method under the current versus proposed case definitions, and to make a 
recommendation on optimal case identification procedures. 

Three key concepts involved in the evaluation of DAWN’S screening systems are the 
concepts of sensitivity, specificity, and positive predictive value. Sensitivity is the probability that 
if a record is a DAWN case it will actually be flagged by the screening process. It is the ratio of 
DAWN cases that screen positive (“true positives”) to all DAWN cases. If the sensitivity of the 
screen is low, many DAWN cases are falsely ruled out by the screening process, and one can 
say that there are a high number of “false negatives.” The specificity of a screen is the ratio of 
non-DAWN cases correctly ruled out to all non-DAWN cases. When specificity is low, there are 
many non-cases that are positively flagged by the screen. In other words, there are a high 
number of “false positives.” 

While sensitivity and specificity of a screen are measures of accuracy, the actual usefulness 
of a screen can depend on the prevalence of the issue being detected. An additional parameter, 
the positive predictive value, is needed to evaluate a screening system. The positive predictive 
value is the probability that a record flagged by the screen will, after medical record review, prove 
to be a DAWN case. It is the ratio of true positive cases to all positively flagged records. For 
DAWN, if the positive predictive value is low (i.e., there are many “false positives” relative to “true 
positives”), DAWN reporters would need to review a high number of records relative to the 
number of DAWN cases actually found. Thus, low positive predictive values increase the burden 
on the reporter. 

An ideal screen has high sensitivity, specificity, and positive predictive value. In practice, 
sensitivity is often increased at the price of positive predictive value. Screens that are highly 
sensitive tend to capture a wide range of potential cases, and there are a number of false 
positives, resulting in lower positive predictive values. Conversely, screens with high positive 
predictive values tend to be more narrow, focusing on groups of cases that have a very high 
probability of meeting case criteria. Flowever, because of the narrowness of the screen, true 
cases belonging to low probability groups are missed and sensitivity is low (A helpful overview of 
sensitivity, specificity, and positive predictive value can be accessed on the Medical University of 
South Carolina’s website at: http://www.musc.edu/dc/icrebm/sensitivitv.html ; Fleiss, 1981). In 
order to decide whether the log screening process is adequate, DAWN must weigh the cost of 
missing cases (as measured by sensitivity) versus the cost of reviewing additional records (as 
measured by positive predictive value). 

For this limited field test of the ED log and ICD-9-CM screening methods, the redesign team 
visited two hospitals where: those methods are currently used, there was a sufficient volume of 
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cases to permit meaningful data analysis, and staff were agreeable to providing limited access to 
a full complement of medical records. The findings of those field tests are summarized here. 2 


Assessment of the Log Screen Method 


The log screen method was assessed during a three-day site visit to an East Coast 
emergency department. Three members of the research team reviewed 1,032 medical records, 
which were all records available for the period from November 5 to November 20, 2000. Data 
sheets were filled out for all cases that made any mention of an illicit drug or any toxic reaction to 
a substance, including licit drugs or consumer products. The broad scope was needed to 
encompass both the current and proposed case definitions, as well as any future revision of the 
proposed definition. In addition to medical record data, the team collected the following 
information from all entries in the ED log for the study dates: presenting complaint, age, gender, 
and disposition. ED log entries were matched with medical records for analysis. Consistent with 
current data collection agreements with DAWN hospitals, medical record information was 
recorded only for cases meeting DAWN criteria; for non-reportable cases, the research team 
recorded only the patient’s log entry. 

The DAWN reporters’ manual includes a list of presenting complaints often associated with 
DAWN cases. Reporters using a log screening method would begin with this list and modify it to 
better reflect the actual cases seen in their EDs. For the purpose of this assessment, data were 
reviewed and a list of presenting complaints was developed that maximized the sensitivity and 
specificity of the log screen for identifying cases meeting the current definition. That same list of 
complaints was then used to screen the records to identify cases meeting the proposed new 
case definition. Additional analyses assessed the number and type of complaints that would 
need to be added or dropped in order to maximize the sensitivity, specificity, and positive 
predictive value of the screen for the new case definition. A key concern was that, were the 
proposed case definition to be implemented, reporters might continue using the old list of 
presenting complaints in their screening procedure. It was therefore necessary to assess the 
extent to which that would impact identification of new cases, and the degree of effort that would 
likely be necessary to update all screening approaches currently in place system-wide. 

After reviewing all charts and log entries, a modified list of presenting complaints was 
developed for use as a case screening tool. 3 The screen was then applied to the database, and 
charts containing any of the complaints included on that list were flagged. In total, the screen 
successfully identified 41 (68.3%) of the 60 cases meeting the current case definition. Thus, the 
sensitivity of the screening procedure was 68 percent, meaning that it missed 32 percent (19/60) 
of the DAWN cases identified by medical record review. This same screening tool successfully 
screened out 841 (90.8%) of the 926 cases that did not meet the current DAWN case definition. 
In other words, it identified 85 false positives and the specificity of the procedure was 91 percent. 
The predictive value of the screen (the percentage of correctly flagged cases among all flagged 
cases) was 41/126 = 32.5 percent. Thus, if a case was flagged by the list of complaints used, 
about one record out of three actually turned out to be a current DAWN case. These results are 
summarized in Table 4-11. 


2 A detailed description of procedures and analyses is beyond the scope of this chapter. Full task reports on the field tests are 
available from OAS upon request. 

3 Because this was a limited assessment, hospital characteristics are unique, and an explanation of procedures are beyond the scope 
of this report, full details are not presented here. The original task reports, including complete lists of complaints that were 
reviewed, listed in the DAWN-ED reporters' manual, and used in these analyses, can be obtained from OAS upon request. 
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Table 4-11. Sensitivity and Specificity of Log Screen Used in Field Test (Current Case 
Definition) __ 



Screen by Revised Criteria 



Identified by 
Screen 

Ruled Out by Screen 

TOTAL 

Medical 

Record 

Review 

Current 

Cases 

41 

(68% of Current Cases) 

19 

60 

Not Current 

85 

841 

(91% of non-DAWN cases) 

926 

Total Reviewed 

126 

860 

986 

Admitted* 

20 

202 

222 

TOTAL 

146 

1,062 

1,208 


*Medical records were not available for patients who had been admitted to other units of the hospital; the 
implications of this missing information are discussed below. 


Next, the same screening tool was applied in an attempt to identify cases meeting the 
proposed new case definition. Of the medical records reviewed, 91 met the new case definition 
criteria while 895 did not. The screening method successfully identified 74 (81.3%) of the 91 
new DAWN cases. There were 17 false negatives (i.e., cases meeting the new definition but not 
flagged by the screen). The sensitivity of this procedure was 81 percent, meaning that the log 
screen missed 19 percent (17/91) of the DAWN cases identified by medical record review. The 
log screen successfully screened out 535 (59.7%) of the 895 cases that did not meet the new 
case definition. However, it also flagged 360 “false positives,” or cases that should have been 
screened out—the specificity of the procedure was 60 percent. The predictive value of the 
screen was 74/434 = 17.1 percent. A reporter reviewing all of the medical records identified by 
the log screen would have found only 17 percent of them to be reportable to DAWN and 83 
percent to be non-reportable. These findings are summarized in Table 4-12. 

The proposed new definition is more sensitive (i.e., the screen identified 81 percent of the 
new DAWN cases compared to 68 percent of those meeting the current case definition) than the 
performance of the log screen criteria used in this study for the current DAWN case definition. 
However, the log screen used with the new definition proved to be much less specific - that is, it 
inaccurately flagged 40 percent of all non-cases as potentially reportable, compared to a false 
positive rate of only 9 percent for the current case definition. In the hospital serving as the study 
site for this test, a DAWN reporter using a log screen and the current case definition would have 
to review 126 medical records to identify 41 DAWN cases and would have missed 19 DAWN 
cases. Meanwhile, a reporter using a log screen and the proposed new case definition would 
have had to review 434 medical records to identify 74 reportable cases and would have missed 
17 reportable cases. If this hospital were to continue to use a log screen once the case definition 
changes, the reporter’s workload would increase more than three-fold, and the reporter would 
still miss 19 percent of the ED’s reportable cases. 
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Table 4-12. Sensitivity and Specificity of Log Screen Used in Field Test (New Case 
_ Definition) _ 



Screen by Broad Net Criteria 



Identified by Screen 

Ruled Out by 
Screen 

TOTAL 

Medical 

Record 

Review 

New Cases 

74 

(81% of New DAWN 
Cases) 

17 

91 

Not New Cases 

360 

535 

(60% of all not-New 
Cases) 

895 

Total Reviewed 

434 

552 

986 

Admitted* 

20 

202 

222 

TOTAL 

454 

754 

1,208 


* Medical records were not available for patients who had been admitted to other units of the hospital; the 
implications of this missing information are discussed below. 


Assessment of the ICD-9-CM Screening Method 


Testing the ICD-9-CM screening method presented substantial technical challenges. The 
test requires access to a facility’s billing records as well as its medical records. To undertake this 
test in a current DAWN hospital, it would have been necessary to ask a billing clerk to develop 
and run repeated algorithms to screen for DAWN cases meeting the current and new case 
definition criteria. To obtain the volume of cases necessary to produce meaningful statistical 
analyses, a large hospital with many substance abuse cases was needed. To avoid the many 
operational difficulties involved in such a test, the redesign team enlisted the assistance of a 
research team in a large, urban, university hospital. Because of its affiliation with a university 
medical center, the hospital was accustomed to supporting research projects of this nature and 
was able to provide graduate research assistants to conduct the medical record reviews. 

The research team reviewed all records generated in the ED for cases treated in the month 
of October 2000. If a case involved drugs, alcohol, or the inhalation of substances, the 
researchers provided the following data for each case: 

■ A case description copied verbatim from the medical record that described the 
presenting problem, the assessment, consultations, and/or toxicology results 

■ Age 

■ Nature of the drug/alcohol involvement (i.e., the “type of case” classification described 
above) 

■ Substance(s) involved 

■ Diagnoses 

■ Disposition at discharge 
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To ensure consistent application of the case definitions, all cases were coded by the 
redesign team (rather than the university’s research assistants) for whether it met the current 
and/or new case definition. Because the case definition proposal had evolved by the time this 
study was conducted, the new case definition included underage drinking cases (which were not 
included in the log screen test conducted earlier). 

The medical center also provided billing data on all cases seen in the month of October 
2000. The billing data included ICD-9-CM codes for primary and secondary diagnoses. A single 
visit could involve documentation of as many as 15 ICD-9-CM codes. In total, the 5,792 ED 
episodes from October 2000 carried a combined total of 1,919 different ICD-9-CM codes. These 
codes were consolidated into 61 categories for the purpose of these analyses; categories were 
based on ICD-9-CM codes and their sub-codes. The categories were developed so as to cluster 
clinically meaningful conditions together, while attempting to maximize the number of categories 
that might be useful as a drug abuse screening tool. (For example, “drug dependence” and all of 
its associated ICD-9-CM codes were collapsed into a single category for analysis.) A total of 61 
categories were developed. Billing records were matched with medical records, and a database 
was created to support the screen test. The sample was split in half using a random sample 
procedure. The first half of the sample was used to develop a list of predictor codes for the ICD- 
9-CM screen. Frequencies were run on the ICD-9-CM categories to identify any that included at 
least one DAWN case. These categories were then entered into a stepwise forward logistic 
regression analysis of all positive predictors. This process was designed to identify the groups of 
ICD-9-CM codes that best identified the current and new DAWN cases. 

Logistic regression is a statistical technique used to find the best fitting model that describes 
the relationship between a dichotomous dependent variable (in this instance, DAWN-reportable 
versus not) and a number of independent variables. The method of maximum likelihood is used 
to determine values for unknown parameters in a linear model that maximizes the probability of 
obtaining the observed set of data. In this study, logistic regression was used to identify which of 
the 61 groups of ICD-9-CM codes were the best independent predictors of whether a case is 
reportable to DAWN. 

Stepwise logistic regression involves the iterative testing of whether variables that have not 
yet been included in the logistic equation significantly increase the probability of obtaining the 
observed pattern of DAWN cases. Variables are added successively. First the best predictor is 
added to the equation, then the remainder of the variables are tested, and the next best predictor 
variable is added. As multiple variables are added to the equation, the independent predictive 
ability of any one of them can change. In order to ensure that only the best predictors are 
included, variables in the equation are tested after a new variable has been added. Variables in 
the equation that no longer have predictive power are deleted. The procedure of adding and 
testing variables in the equation continues until no additional variables add to the predictive 
power of the equation. More detailed information about logistic regression may be found in 
Flosmer and Lemeshow (1989). 

Stepwise logistic regression usually identifies predictors that are associated with either 
group designated by the dichotomous variable (i.e., DAWN cases and non-DAWN cases). 
Flowever, for this field test, predictors in the equation are those categories that will be used to 
identify medical records that need to be reviewed by reporters. Therefore, the model should 
focus on only those categories that are positively associated with status as a DAWN case. 
Reporters are less interested in a list of codes that rule out DAWN cases. Thus, variables that 
were negative predictors were excluded from consideration. Once the best predictors 
(categories) were identified from an analysis of the first half of the sample, they were used to 
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screen for potential DAWN cases in the second half of the sample. A total of 33 ICD-9-CM 
categories were retained for use in the screening process. A sensitivity and specificity analysis 
of this screening process was then conducted. 

Through a stepwise forward regression of current case status on these 33 categories, it was 
found that five ICD-9-CM code categories independently predicted whether a case met criteria 
for the current definition. These were: drug abuse, alcohol abuse, poisoning by drugs, suicide by 
poisoning, and HIV. After the five predictor categories were identified, no other ICD-9-CM codes 
predicted reportability to DAWN at a level higher than chance (p<=. 10). All else being equal, 
cases carrying one of the 47 ICD-9-CM drug abuse codes were 157 times more likely to meet 
the current case definition. 

To test the sensitivity, specificity, and predictive value of the ICD-9-CM screen for current 
DAWN cases, these five categories were used to screen the second half of the sample for 
reportable cases. Medical record review indicated that there were 68 cases in this portion of the 
sample that met the current DAWN case definition. The five ICD-9-CM categories successfully 
identified 46 (68 percent) of 68 current DAWN cases. The sensitivity of this procedure was 68 
percent, meaning that this screen missed 32 percent of the DAWN-reportable cases (22/68), or 
resulted in 22 “false negatives.” 

The same set of five ICD-9-CM code categories successfully screened out 2695 (95 
percent) of the 2831 cases that did not meet current DAWN criteria. However, this screening 
process resulted in identification of 136 “false positives” (i.e., it failed to screen out these cases), 
and the specificity of the procedure was 95 percent. 

The predictive value of this ICD-9-CM screen was 25 percent, calculated as the percentage 
of correctly flagged cases among all flagged cases (in this instance, 46/182). This means that if 
a case screens positive by the ICD-9-CM code categories specified, there is about one chance in 
four that review of the medical record will reveal that the case is reportable to DAWN. Based on 
these data, a DAWN reporter would have had to review the medical records of 182 cases 
flagged by the screen, and would have found 46 to be DAWN-reportable. The results of this 
analysis are summarized in Table 4-13. 


Table 4-13. Performance of ICD-9-CM Screen (Current Case Definition) 




Screen by ICD-9-CM Criteria 



Identified by Screen 

Ruled Out by 
Screen 

TOTAL 

Medical 

Record 

Review 

Current 

Cases 

46 

22 

68 

Not 

Current 

136 

2,695 

2,831 

Total 

182 

2,717 

2,899 
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To develop and test screening criteria for the new case definition, it was necessary to create 
two additional ICD-9-CM code categories - alcohol abuse by a minor and alcohol abuse by 
persons 21 years of age and older. Only the former category meets the proposed criteria for the 
new case definition. Examination of the frequencies of cases captured by each of the 61 ICD-9- 
CM categories used for the above analyses, plus these two additional categories, revealed that 
there were new DAWN cases associated with 40 different ICD-9-CM code categories. 

It is important to note that all cases under age 21 with an alcohol abuse code met the criteria 
for the new case definition based on a review of their medical records. Accordingly, this category 
was a perfect predictor and was included in the screening criteria. However, this category could 
not be entered into a logistic regression, because there was no variation in the dependent 
variable. The other best predictors were identified via a stepwise forward regression of new case 
status on the remaining 39 ICD-9-CM code categories, using the first half of the sample. These 
analyses identified seven additional categories that independently predicted whether a case met 
criteria for the proposed new case definition. These were: drug abuse, poisoning by drugs, 
adverse drug reactions, HIV, rashes, hives, and drug dermatitis. 

Table 4-14 presents the results of the screening procedure when the eight ICD-9-CM 
categories were applied to the second half of the sample (an independent sample). Medical 
record review indicated that 80 cases met criteria for the proposed new case definition among 
the sample. The screening procedure successfully identified 57 (71%) of the 80 DAWN cases. 
The sensitivity of this procedure was 71 percent, meaning that the screening procedure specified 
by the manual missed 29 percent of the DAWN-reportable cases (23/80), resulting in 23 “false 
negatives.” 

The same screening criteria successfully screened out 2709 (96 percent) of the 2819 cases 
that did not meet the new case definition. However, this resulted in identification of 110 “false 
positives” - that is, the ICD-9-CM code categories failed to screen out these cases. The 
specificity of the procedure was 96 percent. 


Table 4-14. Performance of ICD-9-CM Screen (New Case Definition) 




Screen by ICD Criteria 



Identified by Screen 

Ruled Out by 
Screen 

TOTAL 

Medical 

Record 

Review 

New Cases 

57 

23 

80 

Not Cases 

110 

2,709 

2,819 

Total 

167 

2,732 

2,899 


The positive predictive value of the ICD-9-CM screen for the new case definition was 34 
percent, calculated as the percentage of correctly flagged cases among all flagged cases (in this 
instance, 57/167). This means that if a case screens positive by the ICD-9-CM criteria specified, 
there is about one chance in three that review of the medical record will confirm that the case is 
reportable to DAWN. In this hospital, the DAWN reporter would have had to review the medical 
records of the 167 cases flagged by the screening procedure, and would have found 57 to be 
DAWN-reportable. 
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As shown, the ICD-9-CM screen optimized for this hospital still missed a substantial number 
of cases, although it would reduce the burden of reviewing medical records. Comparing the 
performance of the ICD-9-CM screen criteria for the current case definition with criteria targeting 
the proposed new definition, it is clear that they are quite similar in their ability to detect and rule 
out DAWN cases. In other words, moving from the current case definition to the proposed case 
definition seems to have relatively little implication for the ICD-9-CM screen. Of greater concern 
is the underlying sensitivity and utility of the screen itself. Decisions about whether to continue to 
employ ICD-9-CM screening methods must include consideration of whether undercounts of this 
magnitude are acceptable and whether the resources to develop, assess, and refine the ICD-9- 
CM screen are available. 


Discussion 

A considerable difficulty in conducting the log and ICD-9-CM screening tests was an inability 
to gain access to 100 percent of the medical records for patients treated in the ED during the 
study period. Such difficulty is typical of the problems involved in hospital data collection and 
DAWN’S system. Charts will physically move to different sections of the hospital as patients are 
moved to different units or as clinicians require them. In addition, hospitals have idiosyncratic 
methods for tracking patient flow and storing medical records. DAWN hospitals that collect data 
through screening procedures are particularly idiosyncratic; most are not able to use 100 percent 
medical record review, either because reporters cannot access all of the records, or because 
there are too many charts to review in order to identify a relatively small number of cases. 

DAWN procedures seem to value the collection of data over the systematic application of case 
identification methods. While this may be practical, it introduces complications into data quality 
assessments and estimation procedures. Because there is no systematically documented 
information about the screening processes and the hospitals that employ them, it is difficult to 
know what DAWN is actually collecting, and what is being missed. 

A significant data collection challenge faced by DAWN reporters is gaining access to all ED 
charts. Equally challenging is determining what proportion of charts is inaccessible. The 
development of case identification screening methods does not solve this problem. Screening 
methods may only provide a false sense of security. The development of valid, useful, and 
accurate screening criteria requires access to a known proportion of the ED charts for both initial 
development of the screening tool and periodic review of its performance. Before implementing 
any data collection in a hospital, there must be a careful assessment of whether medical record 
access is adequate. Only then can various data collection strategies be evaluated. Because of 
the difficulty involved in developing, assessing, and refining ICD-9-CM screening, the redesign 
team strongly recommended that alternative data collection strategies be considered. Available 
evidence suggests that DAWN would greatly benefit from a concerted effort to: systematically 
assess reporter access to medical records, evaluate the adequacy of current log and ICD-9-CM 
screening procedures, and develop alternatives to inadequate procedures (including 
replacement of deficient hospitals) where needed. 
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